Apr 16, 2020

A quick and dirty way to monitor data arriving on Kafka

I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. Kafka itself is rock-solid - because I’m using ☁️Confluent Cloud and someone else worries about provisioning it, scaling it, and keeping it running for me. But whilst Kafka works just great, my side of the setup—tshark running on a Raspberry Pi—is less than stable. For whatever reason it sometimes stalls and I have to restart the Raspberry Pi and restart the capture process.

May 13, 2016

Monitoring Logstash Ingest Rates with Elasticsearch, Kibana, and Timelion

Yesterday I wrote about Monitoring Logstash Ingest Rates with InfluxDB and Grafana, in which InfluxDB provided the data store for the ingest rate data, and Grafana the frontend. Mark Walkom reminded me on twitter that the next release of Logstash will add more functionality in this area - and that it’ll integrate back into the Elastic stack: @rmoff nice, LS 5.0 will have APIs exposing metrics too. they’ll be integrated back into Marvel/Monitoring!

May 12, 2016

Monitoring Logstash Ingest Rates with InfluxDB and Grafana

In this article I’m going to show you how to easily monitor the rate at which Logstash is ingesting data, as well as in future articles the rate at which Elasticsearch is indexing it. It’s a nice little touch to add to any project involving Logstash, and it’s easy to do. Logstash is powerful tool for data ingest, processing, and distribution. It originated as simply the pipe to slurp at log files and put them into Elasticsearch, but has evolved into a whole bunch more.

Oct 10, 2011

Instrumenting OBIEE - the final chapter

This article has been superseded by a newer version: Instrumenting OBIEE Database Connections For Improved Performance Diagnostics (Previously on this blog: 1, 2, 3…) Summary Instrument your code. Stop guessing. Make your DBA happy. Make your life as a BI Admin easier. The Problem OBIEE will typically connect to the database using a generic application account. (Hopefully, you’ll have isolated it to an account used only for this purpose - if you haven’t, you should.

Feb 2, 2011

Instrumenting OBIEE for tracing Oracle DB calls

Cary Millsap recently published a paper “Mastering Performance with Extended SQL Trace” describing how to use Oracle trace to assist with troubleshooting the performance of database queries. As with all of Cary Millsap’s papers it is superbly written, presenting very detailed information in a clear and understandable way. (and yes I do have a DBA crush ;-)) It discusses how you can automate the tracing of specific sessions on the database, and requiring the application to be appropriately instrumented.

Dec 6, 2010

Adding OBIEE monitoring graphs into OAS

Introduction This is the third part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage. This final article describes how to bolt on to OAS a simple web page hosting the graphs that you created in part 2, plotting data from OBIEE collected in part 1. The webpage This is just an old-school basic HTML page, with a meta-refresh tag (which note that Chrome doesn’t work with) and img tags:

Dec 6, 2010

Charting OBIEE performance data with gnuplot

Introduction This is the second part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage. This article takes data that part one showed you how to collect into a tab-separated file that looks something like this: [sourcecode] 2010-11-29-14:48:18 1 0 11 0 3 2 1 676 340 0 53 1 0 41 0 3 0 2010-11-29-14:49:18 1 0 11 0 3 2 1 676 0 0 0 1 0 0 0 3 0 2010-11-29-14:50:18 2 0 16 1 4 3 1 679 0 0 0 1 0 0 0 4 0 2010-11-29-14:51:18 2 2 19 1 4 3 1 679 32 0 53 1 0 58 0 4 0 2010-11-29-14:52:18 2 1 19 1 4 3 4 682 0 0 0 1 0 0 0 4 0 2010-11-29-14:53:18 2 1 19 1 4 3 4 682 0 0 0 1 0 0 0 4 0 2010-11-29-14:54:18 2 0 19 1 4 3 1 682 0 0 0 1 0 0 0 4 0 [/sourcecode]

Dec 6, 2010

Collecting OBIEE systems management data with jmx

Introduction This is the first part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage. For some background on OBIEE’s Systems Management component, along with JMX and MBeans, see here and here. The following assumes you know your mbeans from coffee beans and jmx from a bmx. The metric collection is built around the jmxsh tool.

Dec 6, 2010

OBIEE monitoring

Those of you who read my blog regularly may have noticed I have a slight obsession with the OBIEE systems management capability which is exposed through JMX. Venkat has blogged this week about JMX in OBI11g, and it’s clearly a technology worth understanding properly. I’ve recently been tinkering with how to make use of it for monitoring purposes, most recently using JConsole and discussed here. What follows is an extension of this idea, cobbled together with a bit of shell scripting, awk, gnuplot, and sticky backed plastic.

Nov 4, 2010

A Poor Man’s OBIEE EM/BI Management Pack

Folk from Yorkshire are tight, so the stereotype goes. So here’s a cheap-ass way to monitor OBIEE 10g using nothing but the OBIEE built-in systemsmanagement component, the jmx agent, and jconsole (which is part of the standard Java distribution): From here you can also export to CSV the various counters, and then store history, plot it out with gnuplot or Excel, etc. If anyone’s interested let me know and I’ll document a bit more about how I did this, but it’s basically building on previous work I’ve documented around jmx and OBIEE.

Jul 29, 2009

OBIEE performance monitoring and alerting with jManage

OBIEE’s Systems Management component exposes configuration and performance data through Java MBeans. As discussed in other posts these can be be accessed through several different ways: JConsole (see also here) oc4j Windows PerfMon (although I guess this isn’t actually using MBeans/JMX?) saw.dll?perfmon BI Management Pack Since it’s a standard java technology being used we can in theory use anything that is designed for monitoring mbeans via jmx. Doing some Googling I discovered jManage.

Jul 16, 2009

JConsole / JMX

[edit] See this post too [/edit] On an OBIEE server run [sourcecode language=“bash”] nohup obiee/systemsmanagement/runagent.sh & [/sourcecode] and then run jconsole (make sure you’ve set the DISPLAY first if you’re running it from UNIX). NB: if you don’t have jconsole in your path you can search for it: [sourcecode language=“bash”] $whereis jconsole jconsole: /opt/java1.5/bin/jconsole /opt/java6/bin/jconsole</span> [/sourcecode] You should find it under your java/bin directory You should get this kind of connection dialog: Click connect, and the console will launch.