Joe Miller bio photo

Joe Miller

Ops/Engineering. Continuous DevOping at Webscale

Twitter Github

Heroku log drains into Logstash

The first and obvious option for shipping logs from a heroku app to Logstash is the heroku input plugin. However, this requires installing the Heroku gem and deploying the login + password of a Heroku user to your Logstash server(s). At this time it seems that any user given permissions to an app on Heroku has full control. Not good when you just want to fetch logs. Heroku has added more granular permissions via OAuth but the Heroku gem does not support OAuth tokens yet.

Fortunately there’s another option using Heroku’s log drain. Setting up a log drain from Heroku to Logstash is almost as simple as the Heroku input plugin but has the major advantage of not requiring any new users or passwords to be deployed on the Logstash server.

Sensu and Graphite, Part 2

In a previous post I described two methods for routing metrics generated by Sensu clients to Graphite:

Method #1 was simply described for completeness. It is not scalable and shouldn’t be used except for very small workloads. Pipe handlers involve a fork() by sensu-server for every metric received.

At the time I recommended method #2 which was more efficient - Sensu would simply copy the metric from its own results queue to another queue that Graphite would be listening on, since both Sensu and Graphite can talk to RabbitMQ.

Dell OpenManage on Fedora 19

Here’s how to get Dell OpenManage running on Fedora 19. It is a simple process but I could not find any up to date information on how to do it elsewhere.

Sensu Presentation from CentOS Dojo Phoenix

I was invited to speak at CentOS Dojo in Phoenix, AZ recently (May 2013) about the Sensu monitoring framework. I wanted to do something a little bit different than past presentations and try to show some use cases that fit what Sensu can do rather than just do a basic introduction to Sensu.

Speeding up Vagrant with parallel provisioning

Simple hack for speeding up vagrant provisions.

Sensu handler sets

In past articles we have covered some of basics of Sensu handlers. A nice feature we haven’t touched on yet is handler “sets”. Handler sets were added around v0.9.2 and can be quite useful for saving time when modifying your handler.

AMQPcat, a netcat-like tool for messaging fun

If you have read @ripienaar’s excellent series of articles on common messaging patterns you probably noticed a handy CLI tool for working with STOMP queues called stompcat. I looked around for something similar for AMQP brokers but couldn’t find anything quite the same. There is amqp-utils but I had some issues with these and the tools didn’t work quite like I was hoping. So I wrote amqpcat with the idea of providing a similar tool to stompcat.

Available on github and

Sensu and Graphite

Updated December 7, 2013 : I no longer recommend using the approach described in this post. Please read Sensu and Graphite, Part 2 instead.

Re-use Nagios plugins in Sensu for quick profit

In my previous article I mentioned a key strength of Sensu is the ability to re-use existing Nagios plugins. This is a powerful feature of Sensu. Nagios has been around for at least 1000 years according to most recent archaeological discoveries, which means a vast amount of human effort (and capital) has gone into creating Nagios plugins. Being able to leverage this prior effort is a huge win. In this article I’ll demonstrate creating a Sensu check with the check_http Nagios plugin.

Getting started with the Sensu monitoring framework

(5/15/2012) NOTE: This guide has been superseded by the official ‘ Install Guide’ doc on the Sensu wiki. The new process utilizes the simpler Omnibus-style Sensu packages and covers installation on Debian/Ubuntu platforms as well. Please use this guide instead of the instructions below.

Correlating Puppet changes to events in your infrastructure using graphite

Sometimes it is pretty obvious when Puppet changes something in your infrastructure and bad things happen in a big dramatic way. Other times it’s not so obvious. It can be invaluable to be able to correlate changes made by Puppet to other events happening in your infrastructure.

List of statsd server implementations

Statsd is a simple client/server mechanism from the folks at Etsy that allows operations and development teams to easily feed a variety of metrics into a Graphite system. For more info on statsd read the seminal blog article on Statsd “Measure Anything, Measure Everything”.

Network Link Conditioner in Xcode 4.1, Lion

Previously, I wrote a post about using the ‘dummynet’ functionality in Mac OSX’s ipfw(8) firewall to simulate a variety of networking conditions, such as:  bandwidth, packet loss, latency (delay). This is a great feature for testing software under a variety of network conditions but it can be a little tough to use unless you’re comfortable at the command line, or even better, have unix scripting skills since there are multiple commands required to create even simple scenarios.

Then, today I noticed that Apple now includes a new prefPane in Xcode 4.1 and Lion called “Network Link Conditioner” that simplifies all of this, and even includes a few profiles to get you started (eg: “Wifi, Average case”, “3G, Lossy Network”.) Pretty cool feature. Especially useful for iOS developers. Screenshot below.


In an attempt to learn some Groovy and Gradle I wrote an implementation of a Statsd client in groovy.  It’s similar to other statsd clients in other languages and supports the typical increment(), decrement(), and timing() methods.

#monitoringsucks - but it doesn't have to

On May 26, 2011, John Vincent, aka @lusis, started a conversation with a simple tweet that led to an hour-plus chat on IRC freenode between some very smart folks.  The IRC channel was supposed to go away after the chat was over but instead it has survived and is attracting more and more attention.

Alternative skins for Skype 5 and 6 on Mac OSX

The consensus at this point is that Skype 5 on Mac OSX is ugly and wastes a ton of screen space for no good reason (all of this is true, btw.) But, you don’t have to live with the ugliness.  It’s not widely known, but Skype 5 supports skins or “chatstyles”, which are constructed entirely in HTML/CSS/Javascript, so they’re easy to write and hack.

Charlie Sheen plugin for the Hudson/Jenkins CI Server

I built a Charlie Sheen “Persona” plugin for our Hudson continuous integration server at work, which is now available on my github page.  If you’re familiar with the Chuck Norris plugin, then you know what this is all about.

Add a little bit of Sheen-ius to your Hudson/Jenkins server!!  Tiger blood is included.  #winning

Kanbanops - new mailing list

I am a fan of using Kanban boards to organize and visualize the work coming into an infrastructure team, but most of the Kanban resources on the internet are geared towards software development teams.

Recently, a new mailing list was started for the sole purpose of sharing Kanban experiences in operations teams. The list is appropriately named kanbanops and you can sign up, browse the archives, etc, over at yahoo groups:

Collectd-Graphite plugin. Bringing together two great tools

Collectd is a powerful tool for gathering metrics using its wide range of plugins, such as cpu, disk, load, memory, etc.  But there is a lack of good frontend tools for visualizing the data collectd produces.

Graphite is an amazingly powerful tool from Orbitz for visualizing metrics, but there is a lack of tools for gathering host-level stats and sending into graphite.

It would be great if we could leverage the strengths of both tools.

HOWTO: install graylog2 on CentOS 5 with RVM + Passenger

Graylog2 is an open-source self-hosted centralized log management tool. Think of it as a do-it-yourself version of, or perhaps a simpler alternative to Splunk. Logs are stored in a MongoDB database.

Client support for Amazon S3 multipart uploads (files >5GB)

I recently had a need to upload a few large (11GB) files to S3 for offsite archive purposes. Amazon announced support for objects >5GB back in November 2010, so I had assumed that most S3 clients and libraries had support for it already.  I was wrong.  It turns out that in order to support large files, you need to use the new “multipart upload” feature.  What I found was that many S3 clients do not yet support multipart uploads.

In case this helps save anyone else a few hours of time, here is a summary of what I found regarding multipart upload support in the various S3 clients:

Amazon Cloudfront Private Streaming CLI Tools

Back in late 2010 I had a need to create a private Cloudfront streaming distribution but there was no simple way to do this. Amazon’s web-based AWS Management Console did not support this and I could not find any simple CLI tools for doing this.

Luckily, RightScale’s right_aws ruby gem (>= 2.0.0) provides support for the Cloudfront private streaming API.

These tools should make it simple for any linux admin or developer to get started with creating a private streaming distribution on Amazon. - testing DNS servers

Sometimes it might be nice to be able to test a DNS server’s output, such as with a continuous monitoring system, or as a validation tool when migrating to a different DNS server (or service.)

$ -z --file --server
Matches: 9
Mis-matches: 2

To see more examples or download the code, goto:

gitosis-ng – Simple, self-service git repositories

Gitosis-ng – it’s gitosis with some new features to help users work with the git server. Mainly implemented with commands sent via ssh, ex:

shellcmd-agent - Execute any shell command in parallel on multiple machines with mcollective

Use mc-shellcmd to run any command on the nodes in your collective, eg:

Simulate network latency, packet loss, and low bandwidth on Mac OSX

Sometimes while testing you may want to be able to simulate network latency, or packet loss, or low bandwidth. I have done this with Linux and  tc/netem as well as with Shunra on Windows, but I had never done it on Mac OSX.

It turns out that Mac OSX includes ‘dummynet’ from FreeBSD which has the capability to do this WAN simulation.

Free IPv4 + IPv6 DNS hosting

I have been a big fan of’s IPv6 tunneling service for a number of years now, but I was pleasantly surprised to discover a new service from (HE - Hurrican Electric) – IPv6 DNS hosting!  Actually it’s dual-stack DNS hosting, allowing you to serve DNS requests for your domains to both the IPv4 and IPv6 internet.

It is currently in “open beta” status and available to anyone with a (free) ipv6 account, or if you happen to be a customer of the parent company providing this service, HE (Hurricane Electric.)

New writer modules for BIND-DLZ's DNS Performance Testing Tools

The BIND-DLZ project publishes an excellent set of performance testing toolswhich make it easy to generate a lot of fake DNS data for a variety of DNS server types.

I have extended this excellent tool set by creating a few new “writer” modules:

RRDTool module for PHP

This is a module for php4 that I wrote a very long time ago that provides access to the rrdtool C library to php scripts.