What's up with Monitorama EU

2016-10-26 23:31:31 by jdixon

Way back in 2013, I put on a small event in Boston, Massachusetts focused on monitoring software and related themes. The unexpected popularity of the show led me to immediately turn around and announce a second event to take place later that year in Berlin, Germany. Both of the conferences were hugely successful and the subject matter clearly resonated with the larger DevOps and Engineering communities.

Read the rest of this story...

Benchmarking Graphite on NVMe

2016-09-13 13:56:30 by jdixon

Here's another quick update to demonstrate what's possible with a single Graphite node running master (these Carbon and Graphite-Web commits, specifically). As you'll see in the results below, this configuration was able to achieve 300k datapoints per second.

This test was performed on a Packet type 3 server with the pair of NVMe flash drives striped in a single LVM volume. Installation of the Graphite stack was still performed using Synthesize v.2.4.1. To take advantage of the increased I/O capacity I added more cache processes for a grand total of eight (8) relays and sixteen (16) caches. Five instances of Haggar ran concurrently, on a separate Packet type 1 server in the same Parsippany, NJ datacenter.

Read the rest of this story...

Benchmarking Graphite master on AWS

2016-09-12 12:15:25 by jdixon

Hello, friends. Just wanted to follow up the previous post with a quick update. As I've mentioned publicly, developing and cutting a new release from Graphite's master branch has become a personal and professional priority for me. And while I've become very familiar with much of the code base over the course of writing The Graphite Book and have thrown a lot of traffic at it over the last few months, I hadn't run any significant tests for performance regressions at scale (compared to the 0.9.15 release).

This round of tests used the same configuration and benchmarking processes as before. I neglected to mention this before, but all series of benchmarks started with installing Graphite using the Synthesize setup script. For the previous test I used Synthesize v.2.4.1 to install Graphite 0.9.15 on a 64-bit Ubuntu 14.04 LTS instance in Amazon's EC2 cloud. For this round I went with Synthesize v3.0.0RC2, which targets Graphite's master branch.

Read the rest of this story...

Benchmarking Carbon and Whisper 0.9.15 on AWS

2016-08-25 15:40:17 by jdixon

This is just a quick post to share some recent benchmarking results for a single Graphite 0.9.15 server. The host is a single EBS-optimized EC2 i2.4xlarge instance with a 400 GiB EBS Provisioned IOPS SSD (io1) with a requested 20k Max IOPS.

I'm not going to dive in too deep with the results, but I'll point out that with the following configuration we were able to increase batch writes effectively, resulting in a peak 38 points per update (pointsPerUpdate, averaged across all cache processes). This means that on average, caches were able to flush 38 datapoints from memory to disk with every write request.

Read the rest of this story...

Everybody Loves Graphite

2015-11-05 23:15:04 by jdixon

There was an article published recently - not here, and not to be linked or referenced here directly - proposing that "nobody loves Graphite" anymore. A linkbait title if I've ever heard one. Many folks linked this article to me, almost certainly expecting me to respond in an uproar. And yet, what I find myself really disappointed over is the obvious misrepresentation of fact (as it pertains to Graphite's technical limits) and an almost malicious disregard for the enormous community that uses it and contributes back to its ongoing development.

It's almost as if they're trying to sell you something. Nah, that couldn't be it.

I readily concede that Graphite was not designed for the transient nature of the sort of bleeding-edge containerized, clustering systems that are becoming popular in conference talks and Hacker News (if not in actual use in production, but we'll forgive them this tiny oversight). Admittedly, it takes an expertly skilled engineer to craft a background job for the purpose of removing old cruft. It's not every SysAdmin that knows how to cron, after all.

Read the rest of this story...

Graphite 0.9.14 - the Highlights

2015-11-01 11:11:12 by jdixon

As I mentioned in the previous blog post, we're perilously close to shipping the next Graphite release. Although we typically avoid large, sweeping changes in the stable branch, the long development cycle leading up to this particular release means we have a number of big new features and performance improvements to announce.

Rather than assume everyone will read and understand the significance of all the changes in the 0.9.14 Release Notes, I felt it would be a good idea to touch on some of them here. This collection represents a small handful of highlights among the numerous changes from this release. The sustained level of interest and contributions from the community continue to astound me. I can't thank everyone enough for their continued support of the Graphite project.

Read the rest of this story...

Graphite 0.9.14 - the Phoenix Release

2015-10-27 23:55:39 by jdixon

If you're like most Graphite users, you're probably wondering if and when there will ever be another release for the project. There hasn't been much public activity over the last couple of years, at least outside of GitHub. A lack of corporate sponsorship, in terms of dedicated developer and maintainer hours, means that the project receives attention as volunteers' schedules permit. Speaking solely for myself, I prioritize Graphite development somewhere behind family, work, the Monitorama conference, writing the Graphite book, and "other recreational activities".

Despite the lack of a regular release cycle, Graphite is as popular as ever. The Grafana project is going gangbusters, with Graphite as its priority time-series backend. A variety of new open source projects have cropped up offering high-performance alternatives to the original specification implementations (graphite-web and carbon). New software projects, both commercial and open source, continue to target Graphite API compatibility because of its ubiquity and ease of use. Heck, even those other competing time-series engines are forced to support Graphite-friendly interfaces. In some cases they even outperform their own proprietary ingress methods.

Read the rest of this story...

On Writing the Graphite Book

2015-02-22 23:56:56 by jdixon

I'm writing a book. This may come as a surprise given the lack of content on this blog over the last... entire year of 2014. Nevertheless, I'm pleased to report that the rumors are true and I am in fact writing a book about Graphite.

Three different editors at O'Reilly contacted me over the course of a few years about the possibility of authoring a volume about my favorite Open Source time-series rendering engine. I had significant concerns about the availability of free time I'd have to spend on this project, so I had to turn them down the first couple times. Last year, something finally clicked and I relented. And so, Monitoring with Graphite became a thing.

We've decided to release it as a work in progress, with the Early Release going on sale in December 2014 and an expected official release around June 2015. According to the outline we're almost at the halfway point of the book, so I think it's reasonable to say we're still on schedule.

If you've enjoyed my blog posts, I really think you'll love the book. I've included a healthy discussion around monitoring concepts and the "composable monitoring system", a deep dive into the Graphite components, fully fleshed-out installation processes and tips of the trade, and a helluva lot more. I aim to be as comprehensive as possible while still managing to keep it an entertaining read. Frankly, this is probably the only subject matter that I'll know well enough to write a book about, so I'm not about to let myself screw it up.

I encourage you to grab the Early Release Ebook and provide feedback. Your comments and suggestions (or questions) will continue to fuel the content for the rest of the book. And if you make it out to Monitorama this summer, I'll be happy to sign your tablet or laptop.

Graphite Tip - A Better Way to Store Events

2014-01-05 20:54:03 by jdixon

Graphite is well known for storing simple key/value metrics using the Whisper time-series database on-disk format. What is not well known about Graphite is that it also ships with a feature known as Events that supports a richer form of metrics storage suitable for, well, events. Imagine a place where you could store tagged metrics and additional data relevant to the event (e.g. code snippets, comments, etc). Many folks use NoSQL databases such as HBase for this purpose, and that's a perfectly reasonable approach. However, if you'd like to store these somewhere where they can be correlated with the rest of your Graphite metrics, then Events might be a good fit for you.

Read the rest of this story...

Graphite Tip - Django 1.4 Admin Workaround

2014-01-05 00:39:38 by jdixon

If you're using Graphite with Django 1.4 or newer, you've probably noticed the broken styling on the Admin module. This appears to be an annoyance at worst, but it's ugly nonetheless. I don't have a fix for this yet, but I have a workaround for anyone using Apache with their Graphite web UI.

Read the rest of this story...