Streaming XML messages from IBM MQ into Kafka into MongoDB
Let’s imagine we have XML data on a queue in IBM MQ, and we want to ingest it into Kafka to then use downstream, perhaps in an application or maybe to stream to a NoSQL store like MongoDB.
Analysing network behaviour with ksqlDB and MongoDB
In this post I want to build on my previous one and show another use of the Syslog data that I’m capturing. Instead of looking for SSH attacks, I’m going to analyse the behaviour of my networking components.
Note You can find all the code to run this on GitHub. Getting Syslog data into Kafka As before, let’s create ourselves a syslog connector in ksqlDB:
CREATE SOURCE CONNECTOR SOURCE_SYSLOG_UDP_01 WITH ( 'tasks.
Copy MongoDB collections from remote to local instance
This is revisiting the blog I wrote a while back, which showed using mongodump and mongorestore to copy a MongoDB database from one machine (a Unifi CloudKey) to another. This time instead of a manual lift and shift, I wanted a simple way to automate the update of the target with changes made on the source.
The source is as before, Unifi’s CloudKey, which runs MongoDB to store its data about the network - devices, access points, events, and so on.
Streaming Data from MongoDB into Kafka with Kafka Connect and Debezium
Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong
MongoDB config - enabling replica sets For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured:
Docs: Replication / Convert a Standalone to a Replica Set
Stop Mongo:
rmoff@proxmox01 ~> sudo service mongod stop Add replica set config to /etc/mongod.
Cloning Ubiquiti’s MongoDB instance to a separate server
DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care!
For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere.
