Like most modern data stores, Elasticsearch was built to be deployed in a clustered environment where data is replicated automatically between various nodes (Cassandra is also in this category). Also like most modern data stores, most developers won’t run a cluster of Elasticsearch nodes for their local development. They’ll only run one, and miss a lot of simple bugs that can crop up from a distributed system. Such as (not a complete list):

• The node you’re talking to goes down
• The node you’re talking to is too slow
• The node you’re talking to has become disconnected from the other nodes
• The data you’ve just written hasn’t yet been indexed (Elasticsearch specific)
• And of course, Heisenbugs!

To make my life easier, I created the instant-elasticsearch repository. instant-elasticsearch uses a couple great tools to make it easy to spin Elasticsearch nodes up and down in AWS:

• Vagrant
• Puppet
• The vagrant-aws plugin for Vagrant
• The elasticsearch/elasticsearch Puppet module

Combining these four building blocks with a simple Puppet manifest and a clever Vagrantfile and instant-elasticsearch is a vagrant up away from automatically provisioning a functional and self-discovering Elasticsearch cluster.

Once the cluster is up and running, you will have to do the manual work of copying some hostnames into your application’s configuration files but only because I’ve not yet had a chance to automate that part (shame on me).

The workflow I use for instant-elasticsearch is one where I come into the office in the morning, spin up my entire cluster fresh which takes about 3-4 minutes. After it’s provisioned, I start my work. Throughout the day I might shutdown some instances here and there to test some fault tolerance, but for all intents and purposes the cluster remains mostly “up” until the end of the day. At the end of the day, I make sure all my code is committed, then run vagrant destroy -f to nuke my cluster.

All said and done I might spend a couple of bucks to save myself countless hours of hunting down the subtle application bugs that might occur in production without sufficient testing locally.

Not too bad for 80 lines of Puppet, and 50 lines of Ruby!

The impossible has happened. Not only have I written a plugin for Jenkins, I’ve released it. An event I’ve long avoided has finally come to pass, mostly thanks to the fantastic Ruby plugin development support developed largley by Charles Lowell.

The Vagrant plugin is still fairly new, and “beta” tested. Thus far I’ve modeled it after the fantastic Android Emulator plugin in that it will bring up a Vagrant VM for the duration of the job, and then destroy it afterwards.

As you can see, the Vagrant plugin adds a couple new build steps to the configure page. One thing to note is that the provisioning is operated as a separate step, instead of bundled in with the boot of the Vagrant machine. This is purposeful, to allow any additional set up of the environment prior to running Puppet or Chef.

The plugin does of course expect that you have a functional VirtualBox environment on the slave you’re running the job on, if you don’t, Vagrant will explode with a fantastic stack-trace and probably burn your house down.

In the near future

I’m thinking about bundling vagrant-snap with the plugin to allow fancy snapshotting support, but I’m not sure of a good definite usecase for that just yet.

If there’s anything you’d like to see, please file a request on the GitHub issues page.

You can find the plugin:

• in your Jenkins “Manage Plugins” page :)
• on GitHub
• on the Jenkins wiki

Enjoy!

NOTE: Be sure to grab version 0.1.2 or later!

The demo is from a running development environment I have locally, but once the v0.1.2 version of the plugin is available in the update center, all of this will work for you as well.