In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made. The data came from my home Ubiquiti router, and took two forms:

Continue Reading

System: rmoff@proxmox01:~$ uname -a Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux rmoff@proxmox01:~$ head -n1 /etc/os-release PRETTY_NAME="Debian GNU/Linux 8 (jessie)" rmoff@proxmox01:~$ python --version Python 2.7.9 Following: https://www.confluent.io/blog/introduction-to-apache-kafka-for-python-programmers/ https://github.com/confluentinc/confluent-kafka-python Install librdkafka, which is a pre-req for the Python library: wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main" sudo apt-get install librdkafka-dev python-dev Setup virtualenv: sudo apt-get install virtualenv virtualenv kafka_push_notify source .

Continue Reading

Presented without comment, warranty, or context - other than these might help a wandering code hacker. When using SigV4, you must specify a ‘host’ parameter boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter. To fix, switch conn_s3 = boto.connect_s3() for conn_s3 = boto.connect_s3(host='s3.amazonaws.com') You can see a list of endpoints here. boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request Make sure you’re specifying the correct hostname (see above) for the bucket’s region.

Continue Reading

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Continue Reading