obfuscurity.
hidden in plain sight

I've been reading about taxing the rich and high unemployment and how the middle class is dying for as long as I can remember. What no one seems to be talking about is that these problems are orthogonal, not causal. Raising the taxes on billionaires is not going to buoy the middle class' ability to buy a new car. And neither will lowering taxes on the rich have the adverse affect. We're no more likely to see a trickle down effect from the government raising taxes as we are from lowering them.

What we really should be asking is what can we do to motivate business owners to increase employee wages without regulation? I'm not sure there is a good answer for that. I think this is a systemic problem within our upper class, one tied to a sense of privilege and a lack of personal responsibility to community.

I don't have any answers. I just wanted to get these thoughts down and see what others think.

One of the most common questions I get from Graphite users is how best to name and/or organize metric paths. I don't have an exhaustive list of "best practices" but I'd like to share some basic insights I've accumulated.

Misaligned paths are ok. I used to be tempted to try and keep different paths aligned in order to ease correlation of related targets within a graph. Fortunately there are plenty of helpful aliasing functions (and wildcards) to help tame unruly paths.

Read the rest of this story...

A little over a week ago I released the Tasseo dashboard. The response I got back was nothing short of astonishing. Tasseo is a Graphite dashboard, one of many to have been released in recent months. That fact alone led me to believe it would fly quietly under the radar. I couldn't have been more wrong; Tasseo (pronounced like Casio) tallied over 200 GitHub watchers in the first weekend, and should pass 300 today.

Tasseo was originally developed as a from-the-ground-up reimplementation of the Pulse dashboard we use at Heroku. Pulse has been a tremendously valuable tool for us; unfortunately, it has some drawbacks that make it a challenge to maintain.

Read the rest of this story...

This morning I was collecting some graphs for one of our weekly status meetings. Asked to find something that represented the state of our Graphite system, I naturally gravitated to my usual standbys, "Carbon_Performance" (top) and "Carbon_Inbound_Bandwidth" (bottom).

1-day	1-week

The SysAdmin in me loves these because they highlight resource utilization on the server. While the former details disk I/O and CPU, the latter tracks inbound bandwidth in terms of bits and packets per-second. Although the network graph seems utterly boring (in as much as we've all used these in one form of another, from vendor-supplied dashboards to Cacti installations), it's this one that is actually the more complicated of the two to configure.

Read the rest of this story...

Let's say you want to compare how a particular metric compares to some point in the past. This is a common practice in troubleshooting and capacity planning. What's the best way to achieve this in Graphite?

I might start off by selecting the past four weeks and visually discern the trends from week to week. Here's a graph showing the last month of AMQP activity. We can see that traffic was oscillating quite a bit over the first week and a half before smoothing out and gradually trending downward.

Read the rest of this story...

I love that Graphite can support per-second resolution. We've started to use it more frequently with applications that emit a constant stream of metrics to one of our aggregators. But there are times when an application might send updates less frequently, or when transient failures or network congestion result in lost metrics. In this case it makes sense to adjust your xFilesFactor value.

You may remember my last post that mentioned the whisper-info.py utility. It helps you extract metadata from your whisper files. Take for example, a whisper file for one of our collectd metrics:

$ sudo whisper-info.py /data/whisper/collectd/63694/swap/used.wsp

maxRetention: 31536000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 534580

Archive 0
retention: 86400
secondsPerPoint: 60
points: 1440
size: 17280
offset: 52

...

Read the rest of this story...

If you've mucked around with your Whisper storage policies or needed to migrate your data to/from Graphite, there's a good chance you've used some of the bin scripts like whisper-info.py and whisper-fetch.py. Unfortunately there are some drawbacks with whisper-fetch.py, most notably that it only fetches content from the first archive to match the requested time period, and it won't return the original raw data after the rollup policies take effect.

Enter whisper-dump.py, a contributed script from Amos Shapira (@amosshapira). This tool dumps the original data for all archives (storage retention periods) as well as the metadata normally retrieved using whisper-info.py.

$ whisper-dump.py elapsed.wsp > dump

$ head -25 dump
Meta data:
  aggregation method: average
  max retention: 31536000
  xFilesFactor: 0.5

Archive 0 info:
  offset: 40
  seconds per point: 1
  points: 300
  retention: 300
  size: 3600

Archive 1 info:
  offset: 3640
  seconds per point: 60
  points: 525600
  retention: 31536000
  size: 6307200

Archive 0 data:
0: 1334532666, 0.0271825
1: 1334532367, 0.0139948
2: 1334532668, 0.0107801
3: 1334532669, 0.0124356
4: 1334531470, 0.0185704

If you're logged into Graphite as an authenticated user you have the option of saving graphs, which will appear under the "My Graphs" folder in the navigation tree to the left. There are some limitations (you can't include spaces in the filename) but it's otherwise a useful feature for saving and sharing graphs with others.

Unknown to some users, Graphite's dot-delimited naming schema is not only available in metrics, but in saved graph names as well. Once you've created or modified a graph, click the Save button (floppy disk icon)...

Enter a dot-delimited path prefix to create a folder. In the following example we're creating a folder named nginx with a graph named conntrack. For additional nested folders, add more path segments.

Graphite will save the graph metadata to the database and popup a dialog confirming the action.

Reload the Graphite window and you'll see the new directory structure.

We use Campfire extensively at $DAYJOB. As our Ops team is 100% remote, it's become indispensable for us. Although it has some minor warts (lack of proper timestamps) it works quite well as a chat medium and collaboration tool. Because of its popularity, there are tons of plugins available. Not the least of which is Hubot, a bot written by GitHub specifically for Campfire.

Hubot supports a wide variety of commands, from useful ones that retrieve Google Maps and Images, to more frivolous examples like PugMe. So it was only a matter of time before I saw the potential for Hubot to be used as a mechanism for sharing Graphite data. After dragging my feet for months, I'm proud to bring you the Graphite script for Hubot.

The usage is simple enough; it only does two things. You can search for saved graphs, returning a list of dot-delimited graph identifiers. It does this by recursively crawling the composer graph directory and returning any matches. Pass one of these names to show and it will display the matching graph in your Campfire room.

Attentive readers might wonder why the bot above is named Merman. This is a private fork of Hubot that's been slightly modified for our own tastes. Never fear, this plugin should work fine with the mainstream Hubot.

I have more interesting ideas for this bot, time permitting. For now, give it a spin and let me know what you think. Pull requests are always welcome.

I remember one day when I was trying to narrow down an application causing high load on an outlier within a fleet of servers. Nagios wasn't suitable for the task, as it only told me which hosts were currently spiking, not which ones have been spiking for a certain window of time. And it certainly couldn't identify a particular host based on a performance visualization.

My Graphite wizard hat went on and I went to work, narrowing down the list of suspects using wildcards and visually inspecting each host's load profile. Within 5 minutes I found my suspect and basked in my glory.

Naturally my brilliance was short-lived.

Read the rest of this story...