Lately I’ve fallen in love with a couple of fairly simple but powerful technologies: haproxy and WSGI (web server gateway interface). While the latter is more of a specification (PEP 333) the concepts it puts forth have made my life significantly easier. In combination, the two of them make for a powerful combination for serving web applications of all kinds and colors.

HAProxy is a robust, reliable piece of load balancing software that’s very easy to get started with, For the uninitiated, load balancing is a common means of distributing the load of a number of inbound requests across a pool of processes, machines, clusters and so on. Whenever you hit any web site of non-trivial size, your HTTP requests are invariably transparently proxied through a load balancer to a pool of web machines.

I started looking into haproxy when I began to move Urlenco.de away from my franken-setup of Lighttpd/FastCGI/Mono/ASP.NET to a pure Python stack. After poking around some articles about haproxy I discovered it can be used for virtual hosts as well as simple load balancing. Using a haproxy’s ACLs feature (see Section 7 in the configuration.txt), you can redirect requests to one backend or another. While my “virtual hosting” with haproxy is using the ability to inspect the HTTP headers of inbound requests, you can use a number of different criterion to determine the right backend for serving a request: url matching, request method matching (GET/POST), protocol matching (haproxy can load balance any kind of TCP connection) and so on.

WSGI (pronounced: whiskey) comes into play on the backend side of haproxy, using the eventlet.wsgi module which provides a WSGI interface I can build web applications very quickly, test them and deploy them. When deployed, I can run them as “nobody” in userspace on the server, binding to some higher numbered port (i.e. 8080) and haproxy will do the work routing to the appropriate WSGI process.

Below is a simple haproxy configuration that I’m using to run Urlenco.de and a site for my wedding and many more as soon as I finish them. The section to note is frontend http-in in which the ACLs are defined for the different virtually hosted domains and the conditionals for selecting a backend based on those ACLs.

global
    maxconn         20000
    ulimit-n        16384
    log             127.0.0.1 local0
    uid             200
    gid             200
    chroot          /var/empty
    nbproc          4
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000

frontend http-in
    bind *:80
    acl is_urlencode hdr_end(host) -i urlenco.de
    acl is_wedding hdr_end(host) -i erinandtylerswedding.com

    use_backend urlencode if is_urlencode
    use_backend wedding if is_wedding
    default_backend urlencode

backend urlencode
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8181 cookie Local

backend wedding
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8081 cookie Local

There’s quite a big advertising blitz in San Francisco for the “Book of Eli” movie, which as far as I can tell is another in a line of quasi-religious films (fortunately Tom Hanks doesn’t star in this one). I now see a billboard that’s a derivation of this on my ride home from work:

To be honest, I was already not going to see this movie but their marketing campaign has hammered the final nail in the cross (er, coffin). Perhaps I’m far more woeful of religion after seeing too many documentaries like “Jesus Camp” and watching clips of Pat Robertson.

In a world of spiteful neo-conservatives hijacking public discourse on important issues with nonsense about the gays, abortion and anything else that can be misconstrued as “christian values.” A world where radical sects of Islam kidnap, murder and terrorize; a world where no religion is without blood on their hands, the billboard is technically right.

I think that’s why it bothers the hell out of me.

Since joining Apture, I’ve primarily concerned myself with lower-level backend code and services, including the machines that our site runs on. While not a drastic departure from my role on the server team at Slide, there are a few notable changes, the largest of which being root. Given the size of Slide’s operations team, a team separate from the “server team” (the latter being developers), my role did not necessitate server management only occasional monitoring. Apture is a different can of beans, we’re simply too small for an operations team, so we work with Contegix to maintain a constant watchful eye on our production environment. Self-assigning myself the “backend guy” hat means server maintenance and operations are part of my concern (but not my focus) since the “goings on” of the physical machines will have a direct impact on the performance and level of service my work can ultimately provide to end users.

Last week while planning some changes we can make to our Django-based production environment that will help us grow more effectively, Steven pointed out that we were going to see an influx of usage today (Jan. 4th) given the large number of users returning to the internet after their holiday vacation. Over the weekend I dreaded what Monday would bring, unable to enact any changes to stave off the inevitable in time.

This morning, waking up uncharacteristically early at 7 a.m. PST, bells were already ringing. A 9 a.m. EST spike angered one of our database machines, by the time I got in the office around 8:10 a.m. PST more bells were ringing as the second wave of users once again angered the MySQL Dolphin Gods. With my morning interview candidate already on site, I furiously typed off a few emails to Contegix sounding the alarm, pleading for help, load balancer tweaks, configuration reviews, anything to help squeeze extra juice from the abnormally overloaded machines to keep our desired level of service up. Working with a few of the talented Contegix admins we quickly fixed some issues with the load balancer under utilizing certain machines in favor of others, isolated a few sources of leaked CPU cycles and discovered a few key places to add more caching with memcached(8).

As our normal peak (~9 a.m. PST to around lunchtime) passed, I started to breathe easier when alarms went of again. Once again, Contegix admins were right there to help us through one of our longest peak I’ve seen since joining Apture, 5:30 a.m. until around 4 p.m.

Survival was my primary objective waking up today but thanks to some initiative and good footwork by the folks at Contegix we not only survived but identified and corrected a number of issues detrimental to performance and discovered on of the key catalysts of cascading load: I/O strapped database servers (as MySQL servers starve for disk I/O, waiting requests in Apache drive the load on a machine through the roof).

I am admittedly quite smitten with Contegix’s work today, I became quite accustomed to KB and his ops team at Slide fixing whatever issues would arise in our production environment and it’s comforting to know that we have that level of sysadmin talent at the ready.

A picture is worth a thousand words; here’s a cumulative load graph from our production Ganglia instance:

While I’m not aggregated into the Python Planet I wanted to join in the meme that’s already going on.

What’s the coolest Python application, framework or library you have discovered in 2009?

While I didn’t discover it until the latter half of 2009, I’d have to say eventlet is the coolest Python library I discovered in 2009. After leaving Slide, where I learned the joys of coroutines (a concept previously foreign to me) I briefly contemplated using greenlet to write a coroutines library similar to what is used at Slide. Fortunately I stumbled across eventlet in time, which shares common ancestry with Slide’s proprietary library.

What new programming technique did you learn in 2009?

I’m not sure I really learned any new techniques over the past year, I started writing a lot more tests this past year but my habits don’t quite qualify as Test Driven Development just yet. As far as Python goes, I’ve been introduced to the Python C API over the past year (written two entire modules in C PyECC and py-yajl) and while I wouldn’t exactly call implementing Python modules in C a “technique” it’s certainly a departure from regular Python (Py_XDECREF I’m looking at you)

What’s the name of the open source project you contributed the most in 2009? What did you do?

Regular readers of my blog can likely guess which open source project I contributed to most in 2009, Cheetah, of which I’ve become the maintainer. I also authored a number of new Python projects in 2009: PyECC a module implementing Elliptical Curve Cryptography (built on top of seccure), py-yajl a module utilizing yajl for fast JSON encoding/decoding, IronWatin an IronPython-based module for writing WatiN tests in Python (supporting screengrabs as well), PILServ an eventlet-based server to do server-side image transformations with PIL, TweepyDeck a PyGTK+ based Twitter client and MicroMVC a teeny-tiny MVC styled framework for Python and WSGI built on eventlet and Cheetah.

What was the Python blog or website you read the most in 2009?

The Python reddit was probably the most read Python-related “blog” I read in 2009, it generally supercedes the Python Planet with regards to content but also includes discussions as well as package release posts.

What are the top three things you want to learn in 2010?

• Python 3. After spending a couple weekends trying to get Cheetah into good working order on Python 3, I must say, maintaining a Python-based module on both Python 2.xx and 3.xx really feels like a nightmare. py-yajl on the otherhand, being entirely C, was trivial to get compiling and executing perfectly for 2.xx and 3.xx
• NoSQL. Earlier this very evening I dumped a boatload of data out of PostgreSQL into Redis and the resulting Python code for data access using redis-py is shockingly simple. I’m looking forward to finding more places where a relational database is overkill for certain types of stored data, and using Redis instead.
• Optimizing Python. With py-yajl Lloyd and I had some fun optimizing the C code behind the module, but I’d love to learn some handy tricks to making pure-Python execute as fast as possible.

A few months ago Kohsuke, author of the Hudson continuous integration server, introduced me to the concept of the “pre-tested commit”, a feature of the TeamCity build management and continuous integration system. The concept is simple, the build system stands as a roadblock between your commit entering trunk and only after the build system determines that your commit doesn’t break things does it allow the commit to be introduced into version control, where other developers will sync and integrate that change into their local working copies. The reasoning and workflow put forth by TeamCity for “pre-tested commits” is very dependent on a centralized version control system, it is solving an issue Git or Mercurial users don’t really run into. Those using Git can commit their hearts out all day long and it won’t affect their colleagues until they merge their commits with others.

In some cases, allowing buggy or broken code to be merged in from another developer’s Git repository can be worse than in a central version control system, since the recipient of the broken code might perform a knee-jerk git-revert(1) command on the merge! When you revert a merge commit in Git, what happens is you not only revert the merge, you revert the commits associated with that merge commit; in essence, you’re reverting everything you just merged in when you likely just wanted to get the broken code out of your local tree so you could continue working without interruption. To solve for this problem-case, I utilize a “pre-tested commit” or “pre-tested merge” workflow with Hudson.

My workflow with Hudson for pre-tested commits involves three separate Git repositories: my local repo (local), the canonical/central repo (origin) and my “world-readable” (inside the firewall) repo (public). For pre-tested commits, I utilize a constantly changing branch called “pu” (potential updates) on the world-readable repo. Inside of Hudson I created a job that polls the world-readable repo (public) for changes in the “pu” branch and will kick off builds when updates are pushed. Since the content of public/pu is constantly changing, the git-push(1) commands to it must be “forced-updates” since I am effectively rewriting history every time I push to public/pu.

To help forcefully pushing updates from my current local branch to public/pu I use the following git alias:

% git config alias.pup "\!f() { branch=\$(git symbolic-ref HEAD | sed 's/refs\\/heads\\///g');\
      git push -f \$1 +\${branch}:pu;}; f"

While a little obfuscated, thie pup alias forcefully pushes the contents of the current branch to the specified remote repository’s pu branch. I find this is easier than constantly typing out: git push -f public +topic:pu

In list form, my workflow for taking a change from inception to origin is:

• hack, hack, hack
• commit to local/topic
• git pup public
• Hudson polls public/pu
• Hudson runs potential-updates job
• Tests fail?
  • Yes: Rework commit, try again
  • No: Continue
• Rebase onto local/master
• Push to origin/master

Using this pre-tested commit workflow I can offload the majority of my testing requirements to the build system’s cluster of machines instead of running them locally, meaning I can spend the majority of my time writing code instead of waiting for tests to complete on my own machine in between coding iterations.

This year my family celebrated the holidays in north Florida at my older sister’s house, fortunately for the location is just as difficult to get to by plane as my parent’s house, so I didn’t have to miss out on any air travel frustration. My trip to north Florida was very boring, my flight out of San Francisco left at 6 a.m. and I arrived in Jacksonville around dinner time (having slept the majority of the flight). The return trip was far more eventful, I left my sister’s house around noon to drive to Jacksonville (roughly an hour and a half trip), waited at the airport for my flight at 4 p.m., arrived in Miami at 5:30, waited for hours on a delayed flight, left Miami around 10 p.m., landed in San Francisco a hours later than anticipated, paid my exorbitant parking fee and sped home.

When I woke up the next day, I looked over at how bitchy and whiney my posts to Twitter from the previous day were. I don’t think I’m normally that big of a jerk but traveling alone, I needed to vent, often. (note: times listed are PST, for the majority of the trip I was in EST)

11:10 AM Still kind of amazes me how many young women in the south are running around with babies in tow.

11:34 AM How the news should cover this incident: Guy tries something on plane, passengers take him out. Post-9/11, TSA is pointless

12:57 PM Shit. My first flight today has propellers. Fucking propellers.

1:22 PM Propeller death trap. http://flic.kr/p/7qzKVn

3:20 PM Step 1 complete; in Miami. Now to SFO.

3:29 PM How does restricting carry-ons make up for the TSA’s incompetence? Predictable kneejerkery.

3:43 PM Waiting on a fucking bus to go back to the terminal. My connection is boarding right the hell now :/

3:56 PM “Fortunately” my flight was delayed an hour, so I didn’t nearly catch it. Fuckin AA.

4:24 PM Let’s just stop flying airplanes, that’s the only real way to stay safe from the terrurists.

4:47 PM It’s okay American Airlines, I really didn’t want to go home anyways.

5:05 PM Outlined a couple posts I want to write on the plane; our original delayed departure time has past, I might never get to writing

6:03 PM More delayment. 2nd potential departure time passed

6:08 PM What kind of toolbox gets luggage embroidered with their initials?

6:11 PM Apparently somebody on the previous flight puked, so we’re delayed while they scrub vomit off the seats. Worth it.

6:26 PM The snacks in Miami are a bit dry http://flic.kr/p/7qHbBN

6:40 PM This plane better make it to SFO, I’ll be pretty pissed to have waited 3 hours to end up in a field.

7:23 PM OMFG I AM TOTALLY ON THE PLANE. 3.5 hours late.

1:45 AM Made it to SFO, only a few hours late. Now to drive home, find a parking spot and cry in the shower a bit

3:05 AM Finally home. Showered, clipped fingernails, q-tip’d ears. Feeling better.

Suffice to say, I don’t think I’m flying American Airlines again for some time (or at all for that matter). The whole experienc to Florida and back was grueling to say the least; with the parking fees, baggage fees, meal fees, delays and endless hours breathing recycled air riddled with H1N1 and sneezes, I think I’m going to keep my feet on the ground for a while.

In a past life I traveled quite frequently, being categorically poor as I often was, I tended to rely on the kindness of friends, family and occasionally total strangers. After breaking the standing record for longest-time-spent-on-Dave’s couch, I came to consider myself a pretty decent house guest. More recently I spent this past week at my older sister’s house with a swarm of other family members, as I cooked breakfast for the family Saturday morning, I decided that I’m not only a pretty decent house guest, I’m a pretty stinkin’ awesome guest who you should invite over if you:

• Feel like cooking a big dinner but don’t want to do the dishes
• Need an expert Rock Band guitarist
• Have children that need entertaining
• Are just sick and tired of cooking and really would like somebody else to make you something delicious
• Feel the need to have a wide-ranging discussion regarding national and international politics with a mildly intelligent person

While I know that you’re not supposed to stay too long as a house guest, I think the social rule comes from a long line of either unwelcome guests or guests that just aren’t doing it right. Here are the rules I try to follow whenever I find myself crashing on some kind person’s couch, floor or air mattress.

Keep your things tidy

Very important, yes you’re likely traveling, the folks you’re staying with understand that you don’t have a closet or dresser you can throw your clothes in, it is very important however that you keep as much of your belongings tidily stashed away in your suitcase. The extra effort goes a long way in making your presence far less impactful on those hosting you. Nobody likes a dirty home.

Offer to cook

Unless your hosts have more money than they should, chances are that they have jobs and when they come home from those jobs they have to cook themselves and their family dinner. Offering to cook goes a long way with a lot of people, especially if you can actually cook. (note: cooking delicious food is not difficult, but really just a test of your ability to read a recipe). Not everybody will take you up on your offer, some people (myself included) find cooking a good way to unwind after a day’s work. If you find yourself in this situation, linger around the kitchen, socialize and try to be as helpful as possible; an extra set of hands and eyes to watch a pot, or peel potatoes is almost always appreciated.

Hang out, don’t cling

Most people enjoy having company, humans are inherently social animals and having a house guest can be an nice change of pace for a lot of people. If you’re traveling through, you’ll have to walk a fine line of hanging out with the hosting party long enough to have fun together but not long enough to make them feel smothered. The system I’ve always followed is to be occupied during the day and social with my hosts in the evening. This gives them a chance to have a normal workday or weekend, and gives me the chance to explore my current location on my own and have an adventure. This set up works quite well when traveling abroad since you get the opportunity to regale your hosts with tales of your adventure in their region over dinner (note; do not trash their city, people tend to have some amount of pride for their city/region/state).

Do chores

If you’re staying with any host for any time longer than a few days, it is highly likely that some cleaning, vacuuming, laundry or dishes will need to be done. A good rule of thumb with chores is not to offer to help but just to help out where you can, a quick “let me give you a hand with that..” will do.

Nobody will turn down a helping hand when it comes to cleaning.

There are occasions when I’ve preferred hotels to crashing with friends or family, when I really need a good night’s rest and some quiet, but if you’re up for a good sociable experience, you really cannot beat crashing on a couch.

Just don’t borrow any money, they really hate that.

After running a Linux laptop for a number of years and having mostly negative travel experiences from messing something up along the way, this holiday season I think I’ve finally figured out how to optimally travel with a Linux notebook. The following tips are some of the lessons I’ve had to learn the hard way through trial and error over the course of countless flights spanning a few years.

Purchase a small laptop or netbook

Far and away the best thing I’ve done for my travel experience thus far has been the purchase of my new Thinkpad X200 (12.1”). My previous laptops include a MacBook Pro (15”), a Thinkpad T43 (14”) and a Thinkpad T64 (14”). Invariably I have the same problems with all larger laptops, their size is unwieldy in economy class and their power consumption usually allows me very little time to get anything done while up in the air. Being 6’4” and consistently cheap, I’m always in coach, quite often on redeye flights where the passenger in front of me invariably leans their seat back drastically reducing my ability to open a larger laptop and see the screen. With a 12” laptop or a netbook (I’ve traveled with an Eee PC in the past as well) I’m able to open the screen enough to see it clearly and actually type comfortbaly on it. Additionally, the smaller screen and size of the laptop means less power consumption, allowing me to use it for extended periods of time.

Use a basic window manager

Personally, I prefer XMonad, but I believe any simplistic window manager will save a noticable number of cycles compared to the Gnome and KDE “desktop environments.” Unlike Gnome, for example, XMonad does not run a number of background daemons to help provide a “nice” experience in the way of applets, widgets, panels and menus.

Disable unneeded services and hardware

Reducing power consumption is a pretty important goal for me while traveling with a Linux laptop, I love it when I have sufficient juice to keep myself entertained for an entire cross-country flight. Two of the first things I disable before boarding a plane are Wireless and Bluetooth via the NetworkManager applet that I run. If I’m on a redeye, I’ll also set my display as dark as possible which not only saves power but also eye strain. It’s also important to make sure your laptop is running its CPU in “power-save” mode, which means the clockspeed of the chip is reduced, allowing you to save even more power. Finally I typically take a look at htop(1) to see if there are any unneeded processes taking up cycles/memory that I either don’t need or don’t intend to use for the flight. The flight I’m currently on (Miami to San Francisco) I discovered that Chrome was churning some unnecessary cycles and killed it (no web browsing on American Airlines).

Use an external device for music/video

If you’re like me, you travel with a good pair of headphones and a desire to not listen to babies crying on the plane. I find a dedicated device purely for music can help avoid wasting power on music since most devices can play for 12-40 hours depending on the device. It’s generally better (in my opinion) to use your $100 iPod for music and your $2000 computer for computing, that might just be personal bias though.

Load applications you’ll need ahead of time

I generally have an idea of what I want to do before I board a plane, I have a project that I’d like to spend some time hacking on or something I want to write out or experiment with. Having a “game plan” before I get onto the plane means I can load up any and all applications while plugged in at the airport. This might be a minor power saver but after I’ve lowered the CPU clockspeed and disabled some services, I certainly don’t want to wait around for applications to load up while I sit idly in coach.

Update: As Etni3s from reddit points out, powertop(1) is a pretty handy utility for watching power consumption.

As I write this article, I’m probably an hour into my five and half hour flight and the battery monitor for my X200 is telling me I have an estimated eight hours of juice left.

I’m proud to say, Tux is my copilot.

Some time ago after reading a post on Eric Florenzano’s blog about hacking together support for Cheetah with Django, I decided to add “proper” support for Cheetah/Django to Cheetah v2.2.1 (released June 1st, 2009). At the time I didn’t use Django for anything, so I didn’t really think about it too much more.

Now that I work at Apture, which uses Django as part of its stack, Cheetah and Django playing nicely together is more attractive to me and as such I wanted to jot down a quick example project for others to use for getting started with Cheetah and Django. You can find the django_cheetah_example project on GitHub, but the gist of how this works is as follows.

Requires

• Django
• Cheetah (>= v2.2.1)

Getting Started

For all intents and purposes, using Cheetah in place of Django’s templating system is a trivial change in how you write your views.

After following the Django getting started documentation, you’ll want to create a directory for your Cheetah templates, such as Cheetar/templates. Be sure to touch __init__.py in your template directory to ensure that templates can be imported if they need to.

Add your new template directory to the TEMPLATE_DIRS attribute in your project’s settings.py.

Once that is all set up, utilizing Cheetah templates in Django is just a matter of a few lines in your view code: import Cheetah.Django

def index(req):
    return Cheetah.Django.render('index.tmpl', greet=False)</code>

Note: Any keyword-arguments you pass into the Cheetah.Django.render() function will be exposed in the template’s “searchList”, meaning you can then access them with $-placeholders. (i.e. $greet)

With the current release of Cheetah (v2.4.1), there isn’t support for using pre-compiled Cheetah templates with Django (it’d be trivial to put together though) which means Cheetah.Django.render() uses Cheetah’s dynamic compilation mode which can add a bit of overhead since templates are compiled at runtime (your mileage may vary).