A few weeks ago I finally tied the knot after a rather long engagement, putting my relationship with then-fianceé into a legally binding relationship. While a wedding should hold a very special place in the bride and groom’s heart forever, I feel like it is safe to say that our wedding objectively rocked.

I don’t want to dive too much into the nitty-gritty details of the entire weekend which culminated in a great ceremony and reception at the phenomenal Madrona Manor Restaurant and Inn. The entire atmosphere, from both families having a great time together, to impeccable weather and the fantastically prepared dinner, was damned near perfect. Cue a brief slideshow of pictures taken by my good friends Dave Young and Annika Lindner:

Now that we’re properly married, and no longer engaged, the typical annoying question has changed from “When are you getting married?” to “When are you having children?”

Considering I can barely take care of our big moron of a cat, I don’t think children are in the cards anytime soon. I’m curious what milestone comes after children though, “when are you going to retire” might be next and then perhaps “when are you going to die” after that.

Either way, I think it’s safe to say, it’s all down hill from here.

The name change that I mentioned in my previous post is now official. This means I now have to update everything. I’m in for a world of hurt between the DMV, banks, brothels and strip-Parcheesi clubs.

The only thing you need to do is update your address book, lucky you! I know at least one friend of mine has, who messaged me to say:

I put your old surname in the “Maiden Name” field in Address Book. Just thought you’d want to know.

I spoke to my step-dad George on the phone immediately after the hearing was over and asked if there are “any perks to being a Croy?”

Still haven’t gotten a response to that one yet.

Tomorrow morning I will be in court, hopefully finalizing a process I started earlier this year. I will be changing my name.

When I was first considering it, I found the entire idea a bit scary. I have worked tremendously hard to make a name for myself, from my work in the open source community to conferences I’ve spoken at and interactions with numerous companies and people who have been instrumental in my whittling out a career in software engineering. I have been very particular about being referred to as “R. Tyler Ballance,” ensuring that my “self-branding” remains consistent, netting me somewhere north of 36,000 results when searching Google.

Tomorrow I intend on throwing all that out the window, there are more important things in life than Google results (as shocking as that may sound).

I’m hesitant to go too much into the motivations for the change, knowing full well that everything I publish might as well be set in stone on the internet.

Those close to me know that my parents divorced when I was young. After a particularly nasty divorce, my mother and my three sisters parted ways with my father who I have since only had sporadic contact with. After a couple dark years for my sisters and I, my mother married another Navy man, George P. Croy, III. George came into the marriage with his daughter, bringing my sister-count up to four.

Over the past fifteen years or so, I have become George’s son. Successfully exploring his emotional spectrum from tears of joy to turning him a bright crimson shade of pissed-off, never once treating me as if I were anything less than his kin. I’m convinced my attitudes towards family, women and friends not to mention my strong opinions on honor and integrity have all been heavily influenced by him

Plainly put, I would not be the man I am today without his guiding hand.

Provided everything goes well at the courthouse, I enter as R. Tyler Ballance and leave as R. Tyler Croy.

Might as well update your address books.

Colleagues of mine are all very familiar with my admiration of Eventlet, a Python concurrency library, built on top of greenlet, that provides lightweight “greenthreads” that naturally yield around I/O points. For me, the biggest draw of Eventlet besides its maturity, is how well it integrates with standard Python code. Any code that uses the built-in socket module can be “monkey-patched” (i.e. modified at runtime) to use the “green” version of the socket module which allows Eventlet to turn regular ol’ Python into code with asynchronous I/O.

The problem with using libraries like Eventlet, is that some Python code just blocks, meaning that code will hit an I/O point and not yield but instead block the entire process until that network operation completes.

In practical terms, imagine you have a web crawler that uses 10 “green threads”, each crawling a different site. The first greenthread (GT1) will send an HTTP request to the first site, then it will yield to GT2 and so on. If each HTTP request blocks for 100ms, that means when crawling the 10 sites, you’re going to block the whole process, preventing anything from running, for a whole second. Doesn’t sound too terrible, but imagine you’ve got 1000 greenthreads, instead of everything smoothly yielding from one thread to another the process will lock up very often resulting in painful slowdowns.

Starting with Eventlet 0.9.10 “blocking detection” code has been incorporated into Eventlet to make it far easier for developers to find these portions of code that can block the entire process. import eventlet.debug eventlet.debug.hub_blocking_detection(True)

While using the blocking detection is fairly simple, its implementation is a bit “magical” in that it’s not entirely obvious how it works. The detector is built around signals, inside of Eventlet a signal handler is set up prior to firing some code and then after said code has executed, if a certain time-threshhold has passed, an alarm is raised dumping a stack trace to the console. I’m not entirely convinced I’m explaining this appropriately so here’s some pseudo-code:

def runloop(): while True: signal.alarm(handler, 1) execute_next_block() if (time.time() - start) < resolution: clear_signal() # Clear the signal if we're less than a second, otherwise it will alarm

The blocking detection is a bit crude and can raise false positives if you have bits of code that churn the CPU for longer than a second but it has been instrumental in incorporating non-blocking DNS support into Eventlet, which was also introduced in 0.9.10 (ported over from Slide’s gogreen package).

If you are using Eventlet, I highly recommend running your code periodically with blocking detection enabled, it is an invaluable tool for determining whether you’re running as fast and as asynchronous as possible. In my case, it has been the difference between web services that are fast in development but slow under heavy stress, and web services that are fast always regardless of load.

I feel like I’m slowly starting to blog like @cansar with just excerpts of other stuff that other people have said on the internet, so this is the last non-technical post for a little bit, promise.

This thread on reddit just about made my morning, well, in addition to that delicious peach I ate.

The mere thought of my own grandfather on reddit or any other online community I frequent is a pretty big stretch, but to have him be a notable member of the community is unfathomable (not to mention, run a part of it like r/mayonnaise).

I suggest you read the whole thread and enjoy a hearty belly laugh, only so long as you’re not doing anything important like driving a bus or performing a colonoscopy.

Updated: As with most things, too good to be true. Although, I must say one of the most well done trolling performances I’ve seen yet. I remain unrepentant in my enjoying of a good belly laugh however

Saving this thread for posterity

Thanks to @pemullen, I was introduced to Sonic.net some time ago. Unfortunately I never took the time in my old apartment to switch out my AT&T DSL for Sonic.net’s Fusion service; the thought of home internet downtime was just too dreadful to even contemplate changing, despite AT&T’s absolutely awful service.

Now that I’ve left that apartment, I can finally take the dive into some delightful Sonic.net service, and while it’s not even installed yet, I can tell this is going to be a wonderful relationship just by some of the support emails I’ve been exchanging with their folks.

From me:

Like an idiot I moved in last weekend instead of this upcoming weekend, so I’m now in the unenviable position of zero home internet service. In the interest of time, can you guys just ship the kit instead of sending some poor tech to Berkeley? :)

I understand that AT&T still needs to install a line, but after that I’m hoping to get up and running as soon as possible, I’m almost to the point of considering opening a book to read.

Oh the horror.

After only a couple hours Kelly R. got back to me:

Sorry to hear that you’ve been driven to such desperate measures. I know the lead time takes a while from AT&T, but we here at Sonic.net have been working on expediting our end of the install process as much as possible. I’ll keep my fingers crossed that this installation process doesn’t result in a library membership.

This fourth of July I find myself thinking a great deal about being an American in the 21st century, and pride. In the back of my head I have that hokey country song “God Bless the USA” with its chorus:

That I’m proud to be an American, where at least I know I’m free. And I wont forget the men who died, who gave that right to me.

The concept comes off so comical to me, “proud to be an American.” What does that even mean? I am no more proud to be an American than I am:

• Proud to have been born in California
• Proud to be white
• Proud to be tall
• Proud to have four sisters
• Proud to have a grandpa named Bob

I had no control in any of it, I won the birth lottery and just happened to be born in the United States. I just happened to have grown up to be a tall, white guy with four sisters and a grandpa named Bob, I didn’t select this configuration, it just happened to me. What’s to be proud of?

Taking pride in one’s country however, I entirely understand. I feel that one should take pride in the positive actions that we undertake as a nation, since it’s actions are theoretically comprised of our collective wills, by the same token, I think one should feel ashamed of the negative actions.

That said, I’m struggling to find things to be proud of America lately, there’s certainly a good bit to be angry and ashamed of:

• Our participating in the secret ACTA treaty negotiations
• The tarring of the Gulf of Mexico, a body of water I’ve spent nearly 40% of my life around.
• Guantanamo
• The shrinking middle class
• Shutting down Shuttle service without a viable Shuttle replacement
• Four horrifically expensive failed wars:
  • Drugs
  • Terror
  • Afghanistan
  • Iraq
• A bloated federal government, with representatives who’ve forgotten who they represent (looking at you Orrin Hatch)
• Irrational fear of nuclear power

I could go on, I could even start a whole new list of all the things we’ve screwed up here in California too, but it just makes angry, then sad, and then sleepy.

I’m sure there are plenty of things that Americans have done lately that one could take pride in, but none are coming to mind.

We have a mighty big hole to dig ourselves out of.

Quite the mixed bag today has been, I went to court (more on that later), I signed a lease (more on that later too), and I worked from home. Since ET and I are leaving this apartment soon, the management company has been showing the apartment during the day. Not a big deal, strangers walk around the apartment, all the windows are opened, all the lights are turned on, doors are opened and closed and if you’re lucky enough to be around, you get to field questions.

About an hour or two after the showing of the apartment was over, ET looks up from the couch and asks “Where’s Buddy” (a.k.a. Fatso). After looking in all of the usual hiding places, she grabs a can of food and taps the lid and listens. A faint meowing is heard. She opens the closet door and taps the lid again. Meow, meow, meow. I think to myself “no way in hell is that cat in the closet, so I hold the can out the window and tap, tap, tap. Meows are coming from outside of the bathroom window. Our bathroom window opens onto this tiny area between two buildings, and is rarely opened because the view sucks, and we don’t stink up our bathroom too much.

Not entirely sure where the cat is, I go to the other side of this little area, in the buildings stairwell and open the window, climb out, and poke around for Fatso, a.k.a Buddy, a.k.a Missing Kitty #1. I can’t see Fatso at all but I can hear him. I tapped on the hood for the ventilation shaft and I hear meowing. I tap again, meowing. Reaching my hand around under the hood, I hear more meowing but I don’t feel anything.

Thanks to a flashlight and mirror loaned from a friendly neighbor, who’s more earthquake prepared than ET and I, I was able to look down the ventilation shaft. and I see Fatso’s stupid little face, all the way at the bottom.

Running down to the basement confirms two things, this cat is stuck, secondly, he’s stuck in the ventilation inlet to the heating system for the building. Stupid cat. While I continue to investigate possible exit strategies, something Fatso clearly hadn’t considered, ET is on telephone duty. First we call the management company, who are characteristically useless, then it’s on to the fire department’s non-emergency line.

When the calvary (see: firemen) arrive, the first thing we do is rip the hood off the ventilation shaft to determine whether we can fish the stupid cat from the depths, which after removing the hood, turns out to be about 15ft. To add insult to injury, there are a couple pieces of wood fastened into place at the top, preventing any beings larger than a 12 pound stupid cat from fitting down the shaft. Looks like we’ll have to attack it from the basement, and be “we” I mean the firemen, I’m useless.

The good boys from the SFFD find a seam in the sheet metal where the shaft attaches to the furnace and using some basic tools (pick) and their hands, are able to tear back some sheet metal so I can poke my head in the bottom of the buildings furnace, only to see our stupid cat, a.k.a Fatso, a.k.a Buddy, a.k.a Missing Cat #1, as far away as possible, entirely unwilling to exit the dark bowels of the furnace he’s occupied for nigh three hours now.

I explain to the firemen, that I can probably handle it from here since they likely have “real shit to do”, but they are unwilling to budge, waiting on “verification” of the cat; they had not actually seen the cat at all up until this point. I shove my head back in the furnace, this time with an arm and grab Fatso by the neck and drag him, against his will, from the furnace to greet the four smiling faces of the SFFD’s finest (and ET).

The firemen are kind enough to seal the now warped sheet metal enough to hold the system over until the management company can repair the damage, and after thanking them they were on their merry way, ideally to save somebody’s life, but most likely to watch Real Housewives of New Jersey back at the station while they wait for something to catch on fire or some stupid cat to poke its head where it doesn’t belong.

Fatso’s favoring his hind-legs a little right now but is all and all in good condition. I want to say he’s learned his lesson, but I’m certain he hasn’t.

In the next few weeks, ET and I will be moving out of San Francisco, perhaps for good. I am living up on the promise I made back in April and leaving. Over the past weekend I was struck by how picky I’ve become, particularly with where I live.

For starters, living in San Francisco, I live in a place with:

• A thriving bicycle culture, which is only looking to get better
• Hundreds of restaurants with all sorts of delicious food stuffs
• Surprisingly few douchebags (hipsters and Mission bartenders not-withstanding)
• Fantastic weather
• Low violent crime

And I’m still not happy with it.

In the past, I’ve lived in places where enormous cars are a status symbol, giant belt buckles that double as shields are accepted; truck nuts. Moving here from Texas I left, stale, windless 100+ degree heat, random people shouting “faggot” at pedestrians from their cars, no tolerance drug policies coupled with binge drinking and drunk driving. To its credit however, Texas is cheap and areas like Austin are wonderful (not counting traffic). When I lived in eastern Germany, I was constantly confused, cold and more than once crashed a bike due to black ice on the roads. Before that, Northern Virginia, living dangerously close to the “south will rise again” group of folks, an area of the country where the Ku Klux Klan is still surprisingly strong, albeit more hidden than before.

Every place that I have lived has had its own unique set of problems, San Francisco included; the lack of progress for a progressive city still irritates the hell out of me.

There are so many parts of this country that unabashedly fucking suck compared to San Francisco, and I’m still not satisfied. What a silly problem to have.

Over the past year or two I’ve become quite fond of tiled window managers, the jump to Awesome (which I’ve since dropped) to XMonad was a logical one. My gratuitous use of GNU/screen and Vim’s tabs and split window support, already provided a de-facto tiled window manager within each one of my many terminals. The tiled window manager on top of all those terminals has served to improve my heavily-terminal biased workflow.

One computer has never been enough for me, at the office my work spans three screens and two computers, I’ve not yet discovered a Thinkpad that can drive three screens alone; at home I typically span three screens and two laptops (let’s conveniently ignore the question of why I feel I need so much screen real estate). Tying these setups together I use synergy to provide my “software KVM” switch. As long as I’ve used synergy, I’ve had to switch from one screen to the other with a mouse, which is one of the few reasons I still keep one on the desk.

Until I discovered a way around that, thanks to Jean Richard (a.k.a geemoo) who posted this little configuration change to synergy.conf:

section: options keystroke(control+alt+l) = switchInDirection(right) keystroke(control+alt+h) = switchInDirection(left) end

With this minor configuration change, combined with XMonad, Vimium (Vim-bindings for Chromium) and my usual bunch of terminal-based applications, I can go nearly mouse-less for almost everything I need to do during the day.

• Hungry


• Too Cold


• Annoyed


• Preparing to jump off the bed


• Jumping off the bed


• Successfully landed on the floor


• Happy


• Welcoming you home


• Appreciating being petted


• Hungry


• Sleepy


• Too Warm


• Unhappy


• Hungry

A couple of weeks ago I wrote a post on how to “Be a Libor”, trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide.

Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a “mental framework” within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).

Libor also had a fantastic quote from the conversation with regards to approaching new problems:

Just because there are multiple right answers, doesn’t mean there’s no wrong answers

Depending on the complexity of the problems you’re facing there are likely a number of solutions but you still can get it wrong, particularly if you don’t remain consistent with your underlying mental framework for the project/organization.

As usual my discussions with Libor are interesting and enjoyable, he’s one of the most capable, thoughtful engineers I know, so I’m interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.

God speed mooncat

Most folks that know me, either virtually or otherwise, know I have opinions. Plenty of opinions, regardless of whether or not I’m qualified to comment on the subject, chances are, I will. At 21st Amendment last Friday, I was in quite a “mood” and poking fun at a few people, of course Can dutifully posted them to twitter, all of which I feel need explaining.

“no, they just apply synergy to paradigms!” - via @cansar

Some how Chris Messina and David Recordon came up in the conversation, I’m not afraid to say that I’ve known of them both for almost three years now, and I still don’t have a clue what they actually do.

“yeah, well he shops at [El] Pollo Loco” - via @cansar

Apparently Can doesn’t know you can buy Bison meat (a tasty alternative to beef), Can also thinks 6 sushi rolls are enough for lunch, suffice to say he has the eating habits of a Maury Povich baby.

“the UK has very lax [child] labor laws, before that he was a chimney sweep” - via @cansar

When discussing Apture’s advisors, Ben Metcalfe came up, smart guy, fun to hang out with but apparently worked for the BBC in his teens, which I didn’t know before Friday evening.

In the interest of full-disclosure, I was drinking.

If you’re interested in hearing me poke fun at myself, you, your startup, your colleagues or your investors, do join me at 21st Amendment next friday at 5 p.m.

Once upon a time I was a Mac developer. I loved Cocoa, I loved building Mac software, Mac OS X was once upon a time the greatest thing ever. I recall writing posts, and even founding a mailing list in the earlier days of Core Data, which I was using in tandem with Cocoa Bindings, which themselves were almost a black art. I was on a couple of podcasts talking about web services with Cocoa or MacWorld. I loved the Mac platform, and would have gladly rubbed Steve Jobs’ feet and thanked him a thousand times for saving Apple from the despair of the late 1990’s. As Apple grew, things slowly started to change, and we started to grow apart.

As I started to drift away, I gave a presentation at CocoaHeads presenting some of the changes and improvements to the Windows development stack, not supremely keen on the idea of building Windows applications, I was clearly on the market for “something else”. Further and further I drifted, until I eventually traded my MacBook Pro in for a Thinkpad, foregoing any future I might have developing Mac software. My decade long journey of tinkering and learning on Macintosh computers had ended.

When Mac OS X was in it’s original Rhapsody-phase, in the weird nether-world between Platinum and Aqua, Apple realized that it had been held back by not giving developers tools to build for the platform. Apple began to push Project Builder which became Xcode, which became the key to the Intel-transition and has helped transform Mac OS from a perennial loser in the third-party software world to a platform offering the absolute best in third-party software. Third-party applications of impressive quality were built and distributed by the “indie mac devs”, Adium, Voodoo Pad and Acorn from Flying Meat, Nicecast and Audio Hijack Pro from Rogue Amoeba, FuzzMeasure Pro from SuperMegaUltraGroovy, Growl, NetNewsWire or MarsEdit originally from Brent Simmons (NetNewsWire is now owned by NewsGator, while MarsEdit was acquired by Daniel Jalkut of Red Sweater Software), Yojimbo and BBEdit from BareBones, even Firefox, Camino and Opera filled the gap while Apple pulled Safari out of it’s craptastic version 2 series. Applications were used on Mac OS X instead of web applications because the experience was better, faster and integrated with Address Book, iPhoto, Mail.app, iMovie and all of Apple’s own stack.

Then came the iPhone, with its “Web SDK” nonsense. The story, at least at the time, was clear to me. Apple didn’t care about me. Apple didn’t care about its developers. Build a web application using JavaScript and AJAX (a Microsoft innovation, I might add) over AT&T’s EDGE network? Fuck you! A number of months later, back-tracking on the “Web SDK” concept, the iPhone SDK came out at WWDC with a ridiculous NDA, forbidding developers from talking about it publicly. Then the App Store was bundled with iTunes and iPhone OS, with Apple becoming the gatekeeper between indie developer, and Joe User. Of course, more recently in the long line of iPhone/developer related tragedies, the infamous Section 3.3.1. There’s also some hub-ub about the Apple Design Awards 2010, only focusing on iPhone and iPad apps which is quite disconcerting for indie mac devs, who routinely compete and win awards for the best Mac applications.

The message is clear, Apple wants to completely own users on its platform and sit between developers and their users, dictating terms.

It’s no wonder that @rentzsch, a major voice in the indie mac dev community, and organizer of the C4 conference is throwing in the towel on organizing C4 entirely (discussed in this post).

It’s not entirely clear whether the “indie mac dev” community will continue to exist for too much longer, there is some speculation that a “Mac App Store” is brewing in Cupertino right now or perhaps modifications to Mac OS X similar to what is present on the iPhone. If I were still part of the “indie mac dev” tribe, I’d feel very nervous right now about what will happen at this year’s WWDC, as Dan Wood from Karelia knows, Apple feels no remorse with stomping on Mac developers.

Worst comes to worst, I sincerely invite indie Mac developers to bring their user-experience talent and software-building energy to the weird but exciting world of web software, so long as Google keeps Facebook in check, the web should remain open for a good long while.

Early this week Can forwarded this post on Quora to me, which asks the question:

Is programming with Twisted really as awful as it sounds?

Yes. Yes. YES IT IS. HOLY CRAP IT’S AWFUL

Here’s some good alternatives:

• Eventlet, my preference
• gevent, an alternative to Eventlet tied to libevent
• Java. because let’s face it, if you’re using Twisted, you’ve already decided not to write Python, so use something with proper threading support.

That is all.

Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro.

Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub.

Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture’s metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro’s RPC support when building services with Eventlet. I’ve talked about Eventlet before, but to recap:

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it

What this means in practice is that you can write highly concurrent network-based services while keeping the code “synchronous” and easy to follow. Underneath Eventlet is the “greenlet” library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or “green threads” whenever a network call blocks.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.

Defining the Protocol

The first part of any Avro RPC project should be to define the protocol for RPC calls. With Avro this entails a JSON-formatted specification, for our echo server example, we have the following protocol:

{"protocol" : "AvroEcho",
"namespace" : "rpc.sample.echo",
"doc" : "Protocol for our AVRO echo server",
"types" : [],
"messages" : {
    "echo" : {
        "doc" : "Echo the string back",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    },
    "split" : {
        "doc" : "Split the string in two and echo",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    }
}}

The protocol can be deconstructed into two concrete portions, type definitions and a message enumeration. For our echo server we don’t need any complex types, so the types entry is empty. We do have two different messages defined, echo and split. The message definition is a means of defining the actual remote-procedure-call, services supporting this defined protocol will need to send responses for both kinds of messages. For now, the messages are quite simple, they expect a query parameter which should be a string, and are expected to return a string. Simple.

(This is defined in protocol.py in the Git repo)

Implementing a Client

Implementing an Avro RPC client is simple, and the same whether you’re building a service with Eventlet or any other Python library so I won’t dwell on the subject. A client only needs to build two objects, an “HTTPTransceiver” which can be used for multiple RPC calls and grafts additional logic on top of httplib.HTTPConnection and a “Requestor”.

client = avro.ipc.HTTPTransceiver(HOST, PORT)
requestor = avro.ipc.Requestor(protocol.EchoProtocol, client)
response = requestor.request('echo', {'query' : 'Hello World'})

You can also re-use for same Requestor object for multiple messages of the same protocol. The three-line snippet above will send an RPC message echo to the server and then return the response.

(This is elaborated more on in client.py in the Git repo)

Building the server

Building the server to service these Avro RPC messages is the most complicated piece of the puzzle, but it’s still remarkably simple. Inside the server.py you will notice that we call eventlet.monkey_patch() at the top of the file. While not strictly necessary inside the server since we’re relying on eventlet.wsgifor writing to the socket. Regardless it’s a good habit to get into when working with Eventlet, and would be required if our Avro-server was also an Avro-client, sending requests to other services. Focusing on the simple use-case of returning responses from the “echo” and “split” messages, first the WSGI server needs to be created:

listener = eventlet.listen((HOST, PORT))
eventlet.wsgi.server(listener, wsgi_handler)

The wsgi_handler is a function which accepts the environment and start_response arguments (per the WSGI “standard”). For the actually processing of the message, you should refer to the wsgi_handler function in server.py in the example repository.

def wsgi_handler(env, start_response):
    ## Only allow POSTs, which is what Avro should be doing
    if not env['REQUEST_METHOD'] == 'POST':
        start_response('500 Error', [('Content-Type', 'text/plain')])
        return ['Invalid REQUEST_METHOD\r\n']

    ## Pull the avro rpc message off of the POST data in `wsgi.input`
    reader = avro.ipc.FramedReader(env['wsgi.input'])
    request = reader.read_framed_message()
    response = responder.respond(request)

    ## avro.ipc.FramedWriter really wants a file-like object to write out to
    ## but since we're in WSGI-land we'll write to a StringIO and then output the
    ## buffer in a "proper" WSGI manner
    out = StringIO.StringIO()
    writer = avro.ipc.FramedWriter(out)
    writer.write_framed_message(response)

    start_response('200 OK', [('Content-Type', 'avro/binary')])
    return [out.getvalue()]

The only notable quirk with using Avro with a WSGI framework like eventlet.wsgi is that some of Avro’s “writer” code expects to be given a raw socket to write a response to, so we give it a StringIO object to write to and return that buffer’s contents from wsgi_handler. The wsgi_handler function above is “dumb” insofar that it’s simply passing the Avro request object into the “responder” which is responsible for doing the work:

class EchoResponder(avro.ipc.Responder):
    def invoke(self, message, request):
        handler = 'handle_%s' % message.name
        if not hasattr(self, handler):
            raise Exception('I can\'t handle this message! (%s)' % message.name)
        return getattr(self, handler)(message, request)

    def handle_split(self, message, request):
        query = request['query']
        halfway = len(query) / 2
        return query[:halfway]

    def handle_echo(self, message, request):
        return request['query']

All in all, minus comments the server code is around 40 lines and fairly easy to follow (refer to server.py for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that.

If you’re curious about some of the other work I’ve been up to with Eventlet, follow me on GitHub :)

I reflect occasionally on how I’ve gotten to where I am right now, specifically to how I made the jump from “just some kid at a Piggly Wiggly in Texas” as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture.

The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave’s team when I joined Slide, and I left Libor’s team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave’s time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I’m at Apture, I’m trying to emulate Libor.

(Note: I do not intend to idolize either of them, but cite important characteristics)

To understand the Libor role, the phrase “the buck stops here” is useful. A Libor is the end of the line for engineering questions, unlike some organizations the “question-chain-of-command” is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you’re screwed.

What does it take to be a Libor you may be thinking:

• No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.

• Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that’s the Libor.

• Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month’s and last year’s short-sighted decisions over and over. A Libor should not let himself nor colleagues “fire and forget” code, poor judgement will haunt a Libor for much longer than most people’s New Year’s resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.

• Focus on Engineering: Code quality and extendability are Libor’s primary focus, that is not to say that a Libor’s role is to impede product development, but rather ensure that it is properly framed. While a product manager’s primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn’t break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.

Playing the Libor character at Apture has been interesting to say the least, I’ve done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.

Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:

Tell me what you want, we’ll find a way to do it

That’s a breakthrough.

I was reading this article on some of the debates going on with the SF Board of Supervisors, where one of the supervisors suggested his intention on bringing a ballot initiative up this November to “force Muni operators to negotiate their salary, benefits and work rules through collective bargaining.” Currently, despite the fact that Muni faces service cuts and a budget shortfall, Muni operators are guaranteed no less than the second-highest transit operator salaries in the nation.

If you ask anybody who knows anything about say, capitalism, economics or even math, they’ll tell you that the arrangement is batshit insane.

So let’s recap, a supervisor suggests the Muni union be forced to use collective bargaining to determine wages, which seems reasonable, and what happens in City Hall?

Daly jumped in to respond to Elsbernd’s comments, saying that if the city wants to save money, it should look at agreements with all unions, beginning with the police. ‘‘Supervisor Elsbernd targeting basically what amounts to a largely black and/or African American union not only has the issue of racial undertones’’

Holy race card Batman! Bargaining, a fundamental tenet of our capitalist society gets one deemed a racist by Supervisor Daly? He’s clearly preoccupied with race, which will sure help bail Muni, and the city, out of their budget deficits this year.

For those of you unfamiliar with San Francisco city budgets, this city somehow pisses away an annual budget of over 6 billion dollars a year, and our streets are terrible, homelessness continues to be a big issue, Muni is a mess and Market St still smells like urine.

When my lease is up in June, I’m leaving San Francisco.

While I wish my friend Ryan all the luck in the world at Blippy, I loved this comment on Reddit regarding the startup from jacques_chester

Blippy is a company which turns the urges of widely available trendoids at the forefront of the hipster web into venture capital investments for its founders.

I still don’t understand what the point is though.