This has already been posted on the Lookout hackers blog but I figured I would repost on my personal blog in lieu of actual content.

My colleague Ian Smith and I will be presenting a talk at JRubyConf EU 2014 titled:

Building a scalable messaging fabric with JRuby and Storm.

The abstract can be found below, but since this is my personal blog I’d like to expand a bit more on why we’re giving the talk.

Last year my team and I deployed ActiveMQ in order to provide an inter-service messaging bus between a new product we were developing and some legacy infrastructure at Lookout. Alongside ActiveMQ we deployed a number of different worker daemons to consume and dispatch messages. These were JRuby-based processes that polled a STOMP connection for new messages and then performed some amount of “work.”

In the process of building out all of this infrastructure we learned a couple of important lessons:

  • ActiveMQ scales like MySQL. That is to say it kind of doesn’t scale, the most effective way to scale out a number of ActiveMQ brokers is to perform some amount of application-layer “sharding” across message brokers. We did not do ActiveMQ any favors in attempting to scale it out effectively, so minus one point us.
  • Many of our problems became less of shoveling work from a single app server into a work-queue, where tools like Sidekiq excel, and more about complex message routing and processing, which Storm was built specifically to solve for.
  • If you provide a good stream of events, everybody, everybody will want to take part in it. One of the unexpected results of deploying ActiveMQ and some messaging facilities was that nearly every team in the company then wanted to take part in the event streams. This wouldn’t have been a problem if we had planned to become the “messaging team” or even had a roadmap to scale out the deployment and managing of messaging infrastructure with ActiveMQ+STOMP. Alas, we did not, and quickly ran into scaling challenges both technically and organizationally.

In our talk, Ian and I will discuss some of these lessons coming from our “old messaging infrastructure” along with some of the important lessons that we’ve learned moving to the newer system. Naturally, neither Storm nor Kafka are silver bullets for scalability, but I believe that they’re the right path to moving towards a broader Lookout-wide messaging fabric.

As previously mentioned, this talk will be at JRubyConf EU which will be held August 1st in Potsdam, Germany. We hope to see you there!

Abstract

As Lookout has grown the number of backend Ruby services the need for reliable, asynchronous service-to-service messaging has gone from “nice to have” to “absolute requirement.”

Our first attempts included some names you may be familiar with: ActiveMQ, Resque, Redis, and Sidekiq. As our infrastructure grew, we found we were reinventing many of the concepts “Storm,” an open source real-time computation/stream-processing system, was specifically designed to handle such as message routing, durability, streaming from multiple inputs, delivering to multiple outputs, etc.

In this talk we’ll cover how we built a scalable service-to-service messaging fabric on top of Storm and Kafka, with JRuby playing a starring role.