Over the past year or two I’ve become quite fond of tiled window managers, the jump to Awesome (which I’ve since dropped) to XMonad was a logical one. My gratuitous use of GNU/screen and Vim’s tabs and split window support, already provided a de-facto tiled window manager within each one of my many terminals. The tiled window manager on top of all those terminals has served to improve my heavily-terminal biased workflow.

One computer has never been enough for me, at the office my work spans three screens and two computers, I’ve not yet discovered a Thinkpad that can drive three screens alone; at home I typically span three screens and two laptops (let’s conveniently ignore the question of why I feel I need so much screen real estate). Tying these setups together I use synergy to provide my “software KVM” switch. As long as I’ve used synergy, I’ve had to switch from one screen to the other with a mouse, which is one of the few reasons I still keep one on the desk.

Until I discovered a way around that, thanks to Jean Richard (a.k.a geemoo) who posted this little configuration change to synergy.conf:

section: options keystroke(control+alt+l) = switchInDirection(right) keystroke(control+alt+h) = switchInDirection(left) end

With this minor configuration change, combined with XMonad, Vimium (Vim-bindings for Chromium) and my usual bunch of terminal-based applications, I can go nearly mouse-less for almost everything I need to do during the day.

• Hungry


• Too Cold


• Annoyed


• Preparing to jump off the bed


• Jumping off the bed


• Successfully landed on the floor


• Happy


• Welcoming you home


• Appreciating being petted


• Hungry


• Sleepy


• Too Warm


• Unhappy


• Hungry

A couple of weeks ago I wrote a post on how to “Be a Libor”, trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide.

Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a “mental framework” within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).

Libor also had a fantastic quote from the conversation with regards to approaching new problems:

Just because there are multiple right answers, doesn’t mean there’s no wrong answers

Depending on the complexity of the problems you’re facing there are likely a number of solutions but you still can get it wrong, particularly if you don’t remain consistent with your underlying mental framework for the project/organization.

As usual my discussions with Libor are interesting and enjoyable, he’s one of the most capable, thoughtful engineers I know, so I’m interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.

God speed mooncat

Most folks that know me, either virtually or otherwise, know I have opinions. Plenty of opinions, regardless of whether or not I’m qualified to comment on the subject, chances are, I will. At 21st Amendment last Friday, I was in quite a “mood” and poking fun at a few people, of course Can dutifully posted them to twitter, all of which I feel need explaining.

“no, they just apply synergy to paradigms!” - via @cansar

Some how Chris Messina and David Recordon came up in the conversation, I’m not afraid to say that I’ve known of them both for almost three years now, and I still don’t have a clue what they actually do.

“yeah, well he shops at [El] Pollo Loco” - via @cansar

Apparently Can doesn’t know you can buy Bison meat (a tasty alternative to beef), Can also thinks 6 sushi rolls are enough for lunch, suffice to say he has the eating habits of a Maury Povich baby.

“the UK has very lax [child] labor laws, before that he was a chimney sweep” - via @cansar

When discussing Apture’s advisors, Ben Metcalfe came up, smart guy, fun to hang out with but apparently worked for the BBC in his teens, which I didn’t know before Friday evening.

In the interest of full-disclosure, I was drinking.

If you’re interested in hearing me poke fun at myself, you, your startup, your colleagues or your investors, do join me at 21st Amendment next friday at 5 p.m.

Once upon a time I was a Mac developer. I loved Cocoa, I loved building Mac software, Mac OS X was once upon a time the greatest thing ever. I recall writing posts, and even founding a mailing list in the earlier days of Core Data, which I was using in tandem with Cocoa Bindings, which themselves were almost a black art. I was on a couple of podcasts talking about web services with Cocoa or MacWorld. I loved the Mac platform, and would have gladly rubbed Steve Jobs’ feet and thanked him a thousand times for saving Apple from the despair of the late 1990’s. As Apple grew, things slowly started to change, and we started to grow apart.

As I started to drift away, I gave a presentation at CocoaHeads presenting some of the changes and improvements to the Windows development stack, not supremely keen on the idea of building Windows applications, I was clearly on the market for “something else”. Further and further I drifted, until I eventually traded my MacBook Pro in for a Thinkpad, foregoing any future I might have developing Mac software. My decade long journey of tinkering and learning on Macintosh computers had ended.

When Mac OS X was in it’s original Rhapsody-phase, in the weird nether-world between Platinum and Aqua, Apple realized that it had been held back by not giving developers tools to build for the platform. Apple began to push Project Builder which became Xcode, which became the key to the Intel-transition and has helped transform Mac OS from a perennial loser in the third-party software world to a platform offering the absolute best in third-party software. Third-party applications of impressive quality were built and distributed by the “indie mac devs”, Adium, Voodoo Pad and Acorn from Flying Meat, Nicecast and Audio Hijack Pro from Rogue Amoeba, FuzzMeasure Pro from SuperMegaUltraGroovy, Growl, NetNewsWire or MarsEdit originally from Brent Simmons (NetNewsWire is now owned by NewsGator, while MarsEdit was acquired by Daniel Jalkut of Red Sweater Software), Yojimbo and BBEdit from BareBones, even Firefox, Camino and Opera filled the gap while Apple pulled Safari out of it’s craptastic version 2 series. Applications were used on Mac OS X instead of web applications because the experience was better, faster and integrated with Address Book, iPhoto, Mail.app, iMovie and all of Apple’s own stack.

Then came the iPhone, with its “Web SDK” nonsense. The story, at least at the time, was clear to me. Apple didn’t care about me. Apple didn’t care about its developers. Build a web application using JavaScript and AJAX (a Microsoft innovation, I might add) over AT&T’s EDGE network? Fuck you! A number of months later, back-tracking on the “Web SDK” concept, the iPhone SDK came out at WWDC with a ridiculous NDA, forbidding developers from talking about it publicly. Then the App Store was bundled with iTunes and iPhone OS, with Apple becoming the gatekeeper between indie developer, and Joe User. Of course, more recently in the long line of iPhone/developer related tragedies, the infamous Section 3.3.1. There’s also some hub-ub about the Apple Design Awards 2010, only focusing on iPhone and iPad apps which is quite disconcerting for indie mac devs, who routinely compete and win awards for the best Mac applications.

The message is clear, Apple wants to completely own users on its platform and sit between developers and their users, dictating terms.

It’s no wonder that @rentzsch, a major voice in the indie mac dev community, and organizer of the C4 conference is throwing in the towel on organizing C4 entirely (discussed in this post).

It’s not entirely clear whether the “indie mac dev” community will continue to exist for too much longer, there is some speculation that a “Mac App Store” is brewing in Cupertino right now or perhaps modifications to Mac OS X similar to what is present on the iPhone. If I were still part of the “indie mac dev” tribe, I’d feel very nervous right now about what will happen at this year’s WWDC, as Dan Wood from Karelia knows, Apple feels no remorse with stomping on Mac developers.

Worst comes to worst, I sincerely invite indie Mac developers to bring their user-experience talent and software-building energy to the weird but exciting world of web software, so long as Google keeps Facebook in check, the web should remain open for a good long while.

Early this week Can forwarded this post on Quora to me, which asks the question:

Is programming with Twisted really as awful as it sounds?

Yes. Yes. YES IT IS. HOLY CRAP IT’S AWFUL

Here’s some good alternatives:

• Eventlet, my preference
• gevent, an alternative to Eventlet tied to libevent
• Java. because let’s face it, if you’re using Twisted, you’ve already decided not to write Python, so use something with proper threading support.

That is all.

Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro.

Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub.

Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture’s metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro’s RPC support when building services with Eventlet. I’ve talked about Eventlet before, but to recap:

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it

What this means in practice is that you can write highly concurrent network-based services while keeping the code “synchronous” and easy to follow. Underneath Eventlet is the “greenlet” library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or “green threads” whenever a network call blocks.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.

Defining the Protocol

The first part of any Avro RPC project should be to define the protocol for RPC calls. With Avro this entails a JSON-formatted specification, for our echo server example, we have the following protocol:

{"protocol" : "AvroEcho",
"namespace" : "rpc.sample.echo",
"doc" : "Protocol for our AVRO echo server",
"types" : [],
"messages" : {
    "echo" : {
        "doc" : "Echo the string back",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    },
    "split" : {
        "doc" : "Split the string in two and echo",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    }
}}

The protocol can be deconstructed into two concrete portions, type definitions and a message enumeration. For our echo server we don’t need any complex types, so the types entry is empty. We do have two different messages defined, echo and split. The message definition is a means of defining the actual remote-procedure-call, services supporting this defined protocol will need to send responses for both kinds of messages. For now, the messages are quite simple, they expect a query parameter which should be a string, and are expected to return a string. Simple.

(This is defined in protocol.py in the Git repo)

Implementing a Client

Implementing an Avro RPC client is simple, and the same whether you’re building a service with Eventlet or any other Python library so I won’t dwell on the subject. A client only needs to build two objects, an “HTTPTransceiver” which can be used for multiple RPC calls and grafts additional logic on top of httplib.HTTPConnection and a “Requestor”.

client = avro.ipc.HTTPTransceiver(HOST, PORT)
requestor = avro.ipc.Requestor(protocol.EchoProtocol, client)
response = requestor.request('echo', {'query' : 'Hello World'})

You can also re-use for same Requestor object for multiple messages of the same protocol. The three-line snippet above will send an RPC message echo to the server and then return the response.

(This is elaborated more on in client.py in the Git repo)

Building the server

Building the server to service these Avro RPC messages is the most complicated piece of the puzzle, but it’s still remarkably simple. Inside the server.py you will notice that we call eventlet.monkey_patch() at the top of the file. While not strictly necessary inside the server since we’re relying on eventlet.wsgifor writing to the socket. Regardless it’s a good habit to get into when working with Eventlet, and would be required if our Avro-server was also an Avro-client, sending requests to other services. Focusing on the simple use-case of returning responses from the “echo” and “split” messages, first the WSGI server needs to be created:

listener = eventlet.listen((HOST, PORT))
eventlet.wsgi.server(listener, wsgi_handler)

The wsgi_handler is a function which accepts the environment and start_response arguments (per the WSGI “standard”). For the actually processing of the message, you should refer to the wsgi_handler function in server.py in the example repository.

def wsgi_handler(env, start_response):
    ## Only allow POSTs, which is what Avro should be doing
    if not env['REQUEST_METHOD'] == 'POST':
        start_response('500 Error', [('Content-Type', 'text/plain')])
        return ['Invalid REQUEST_METHOD\r\n']

    ## Pull the avro rpc message off of the POST data in `wsgi.input`
    reader = avro.ipc.FramedReader(env['wsgi.input'])
    request = reader.read_framed_message()
    response = responder.respond(request)

    ## avro.ipc.FramedWriter really wants a file-like object to write out to
    ## but since we're in WSGI-land we'll write to a StringIO and then output the
    ## buffer in a "proper" WSGI manner
    out = StringIO.StringIO()
    writer = avro.ipc.FramedWriter(out)
    writer.write_framed_message(response)

    start_response('200 OK', [('Content-Type', 'avro/binary')])
    return [out.getvalue()]

The only notable quirk with using Avro with a WSGI framework like eventlet.wsgi is that some of Avro’s “writer” code expects to be given a raw socket to write a response to, so we give it a StringIO object to write to and return that buffer’s contents from wsgi_handler. The wsgi_handler function above is “dumb” insofar that it’s simply passing the Avro request object into the “responder” which is responsible for doing the work:

class EchoResponder(avro.ipc.Responder):
    def invoke(self, message, request):
        handler = 'handle_%s' % message.name
        if not hasattr(self, handler):
            raise Exception('I can\'t handle this message! (%s)' % message.name)
        return getattr(self, handler)(message, request)

    def handle_split(self, message, request):
        query = request['query']
        halfway = len(query) / 2
        return query[:halfway]

    def handle_echo(self, message, request):
        return request['query']

All in all, minus comments the server code is around 40 lines and fairly easy to follow (refer to server.py for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that.

If you’re curious about some of the other work I’ve been up to with Eventlet, follow me on GitHub :)

I reflect occasionally on how I’ve gotten to where I am right now, specifically to how I made the jump from “just some kid at a Piggly Wiggly in Texas” as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture.

The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave’s team when I joined Slide, and I left Libor’s team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave’s time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I’m at Apture, I’m trying to emulate Libor.

(Note: I do not intend to idolize either of them, but cite important characteristics)

To understand the Libor role, the phrase “the buck stops here” is useful. A Libor is the end of the line for engineering questions, unlike some organizations the “question-chain-of-command” is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you’re screwed.

What does it take to be a Libor you may be thinking:

• No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.

• Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that’s the Libor.

• Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month’s and last year’s short-sighted decisions over and over. A Libor should not let himself nor colleagues “fire and forget” code, poor judgement will haunt a Libor for much longer than most people’s New Year’s resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.

• Focus on Engineering: Code quality and extendability are Libor’s primary focus, that is not to say that a Libor’s role is to impede product development, but rather ensure that it is properly framed. While a product manager’s primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn’t break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.

Playing the Libor character at Apture has been interesting to say the least, I’ve done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.

Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:

Tell me what you want, we’ll find a way to do it

That’s a breakthrough.

I was reading this article on some of the debates going on with the SF Board of Supervisors, where one of the supervisors suggested his intention on bringing a ballot initiative up this November to “force Muni operators to negotiate their salary, benefits and work rules through collective bargaining.” Currently, despite the fact that Muni faces service cuts and a budget shortfall, Muni operators are guaranteed no less than the second-highest transit operator salaries in the nation.

If you ask anybody who knows anything about say, capitalism, economics or even math, they’ll tell you that the arrangement is batshit insane.

So let’s recap, a supervisor suggests the Muni union be forced to use collective bargaining to determine wages, which seems reasonable, and what happens in City Hall?

Daly jumped in to respond to Elsbernd’s comments, saying that if the city wants to save money, it should look at agreements with all unions, beginning with the police. ‘‘Supervisor Elsbernd targeting basically what amounts to a largely black and/or African American union not only has the issue of racial undertones’’

Holy race card Batman! Bargaining, a fundamental tenet of our capitalist society gets one deemed a racist by Supervisor Daly? He’s clearly preoccupied with race, which will sure help bail Muni, and the city, out of their budget deficits this year.

For those of you unfamiliar with San Francisco city budgets, this city somehow pisses away an annual budget of over 6 billion dollars a year, and our streets are terrible, homelessness continues to be a big issue, Muni is a mess and Market St still smells like urine.

When my lease is up in June, I’m leaving San Francisco.

While I wish my friend Ryan all the luck in the world at Blippy, I loved this comment on Reddit regarding the startup from jacques_chester

Blippy is a company which turns the urges of widely available trendoids at the forefront of the hipster web into venture capital investments for its founders.

I still don’t understand what the point is though.

As I sit here writing, burnt out from the incredible hype surrounding Apple’s iPad launch and I cannot help but have flashbacks. Flashbacks to 2004/2005 when “Web 2.0” was at its peak, “ajax” and “mashup” were quite trendy. Flashbacks to mid-2007 when the Facebook Platform was gaining steam, words like “FBML” and “social applications” became the new lingo. Flashbacks to early 2008 when the iPhone SDK launched and hoards of developers rushed to submit their apps to the App Store. All bore a resemblance to today, with the iPad, the newest, hottest thing in the world with everybody and their mother vying for one.

Every step of the way, beneath the almost overwhelming marketing and hype, lies actual technological innovation. With “Web 2.0”, the underlying core innovation was the rise of the web as a formidable platform, browsers became capable of supporting immersive applications built in JavaScript, the web “came alive”, no longer a lexicon of static pages. With the launch of the Facebook Platform, products could engage not just on a one-to-one level with a user, but with a user’s social circle. The iPhone SDK made developers re-think building applications, touch, motion and then location became fundamental building blocks for products. The iPad represents a slightly different innovation, the introduction of “casual computing” to the masses. Personal computers are becoming omnipresent, the smartphone in your pocket, the laptop on your desk and now the tablet on the coffee table.

It comes as no surprise that just as with every other hyped innovation over the past few years, there’s a rush of gold-diggers trying to latch onto somebody else’s innovation, using it to rocket them to the top. The announcement of a $200m “iFund” for iPad apps reminds me a lot of the crazy venture capitalists establishing Facebook app funds or even the $100m “iFund” for iPhone apps (from the same group no less).

Venture Capitalists aren’t alone in their attempts to ride Apple’s innovating coat tails, companies building products are falling for it too, with hundreds of thousands of apps in the iPhone App Store or the hundreds of thousands of Facebook applications, one could deceive themselves into believing tens of thousands of companies are printing money on these platforms!

I would estimate that for every 1000 applications, on any of these platforms, only one is a good product. The same thing is going to happen on the iPad, for the exact same reason that it occurred with Web 2.0 sites, the Facebook Platform and the iPhone.

It’s like X, but for the Web / Facebook Platform / iPhone / iPad!

Platform innovation is no substitute for product innovation. The iPad app for the New York Times comes to mind, which apparently does nothing but format the New York Times and show the user interstitial ads. There is really no added value over using something like NetNewsWire or god forbid, browsing to newyorktimes.com with Safari for consuming content.

Supporting the iPad is not a product, it’s a feature, if your product isn’t already compelling there is nothing intrinsic about the iPad that will make it more compelling just as it was with the launch of the other platforms.

While I won’t be buying an iPad, I wish all those jumping on the bandwagon luck. With a word of caution, bring features because your presence on the iPad alone will not be enough.

Being a big fan of “continuous integration”, particularly with Hudson, I’ve often thought about the possibilities of turning it into a business. It’s no surprise really, my first commercial application as a rogue Mac software developer was a product called BuildFactory which, while fun to build, never sold all that many licenses. With the advent of Amazon’s EC2 service and the transition of these cloud computing resources into a building block for many businesses, I’ve long thought about the idea of building “continuous integration as a service.”

At face value the idea sounds incredibly fun to build, I’ll build a service that integrates with GitHub, Google Code, SourceForge and private source control systems. The end (paying) user would “plug-in” to the “continuous integration grid”, they’d work throughout the day, committing code and then the CI grid would pick up those changes, build releases and run tests against a number of different architecture, automatically detecting failures and reporting them back to the developers. It involves some of my favorite challenges in programming:

• Scaling up
• Efficiently using cycles, and only when needed
• Building and testing cross-architecture and cross-platform

Unfortunately, it’s a crap business idea, I now have second-hand confirmation from a group of guys who’ve attempted the concept. The folks behind RunCodeRun are shutting down the service. In the post outlining why they’re shutting down, they’ve hit the nail on the head on why “continuous integration as a service” can never work:

Large scale hosted continuous integration is consumed as a commodity but built as a craft, and the rewards, both emotional and financial, are insufficient to support the effort.

Elaborating further on their point, continuous integration by itself is a relatively basic task: build, test, repeat. The biggest problem with continuous integration as a service however, is that no two projects are alike. My build targets or requirements might be vastly different from project to project, let alone customer to customer, making the amount of tweaking and customization per-job too large such that at some point the only benefit that one derives from such a service is the hosting of the machines to perform the task. If you’re just taking care of that, why wouldn’t your customers just use Hudson in “the cloud” themselves? The CI grid at that point offers no exceptional value.

As much as I regret letting a fun idea die, I think I’ll have to file this one under “To do after becoming so rich I’ll care about capital gains taxes.”

Typically I read at least one news story a day that irritates me, usually I either don’t care enough to gripe about them further, or I forget. After griping at ET about driving in the car with a phone in her hand, I remembered an article I read the SF Streets Blog titled: “Advocates Concerned That Cyclists Are Included in Distracted Driving Bill” (link)

One of the choice quotes from the article being:

The California Bicycle Coalition (CBC), which was an early supporter of the original distracted driving legislation, was not thrilled about the inclusion of cyclists in the bill. CBC Communications Director Jim Brown said that he was confused about the motivation for extending the same level of fines to cyclists, particularly absent data showing distracted cycling as a public safety hazard.

“The consequences of a distracted driver are considerably more serious than the consequences of distracted cycling,” said Brown, adding that safe riding should be encouraged at all times and that talking on a cell phone or any other practice that distracted a cyclist from riding would not be advisable.

As a member of the San Francisco Bicycle Coalition, I’m partially annoyed by Mr. Brown’s comments, but I don’t particularly care. Reading further through the article, I found this:

Andy Thornley, Program Director for The San Francisco Bicycle Coalition, agreed with Winter that lumping cyclists with motorists in this law was not good policy. While the SFBC “teaches and preaches safe, respectful, and mindful bicycling,” said Thornley, “we’re very leery of any equivalence of penalty when punishing a guilty cyclist or driver for the same offense.”

“Even worse, we wonder whether bicyclists would be cited more often than motorists because it’s so much easier to spot someone texting while pedaling,” he added. “It’s already a problem of perception that individual bicycle riders seem to be noticed being naughty more than motorists, comfortably anonymous within their glass and steel boxes.”

What a hypocrite! Riding your bike while on the phone or worse, texting is just as stupid as some of the no-helmet, no-light nonsense I was incensed over a few weeks ago, but the fact that these two gentlemen from Bike Coalitions want preferential treatment for cyclists in the most idiotic way possible blows my mind. To be honest, I’m entirely in favor of bicyclists being cited more often than motorists for breaking the law (running red lights or not using signals comes to mind).

This kind of no-distractions law makes a lot of sense to me, and should be applied to just about anybody operating a moving vehicle, bikes, trikes, motorcycles, mopeds, cars, tractors, law mowers, you name it. If you are operating a vehicle distracted you raise your chances of hurting yourself or others on public roads (ever been hit while walking by a cyclist?).

Inside of the San Francisco cycling community, I think we can do our part by shunning or otherwise pushing cyclists into light posts who are on their cell phones while riding. They are clearly morons and in my opinion the CBC or the SFBC has no place defending their idiocy.

Writing software is an outlet for artistic expression to many people, myself included. For me, solving problems involves a good deal of creativity not only in the actual solution but also in the manipulating several moving parts in order to fit the solution into an existing code-base. Combining this creative outlet with a beautiful language, such as Python results in some developers writing code that holds an masterpiece-level of beauty to them, to the untrained eye one might look at a class and think nothing of it, but to the author of that code, it might represent a substantial amount of work and personal investment.

Like art, sometimes the beauty is entirely subjective. there has been times where I’ve been immensely pleased with one of my creations, only to turn to wholly unimpressed Dave. Managing or working with any team of highly motivated, passionate and creative developers presents this problem, as a group: how can you objectively judge code while preserving the sense of ownership by the author? The first step to objectively judging code in my opinion, is to separate it from the individual who wrote it when discussing the code. For a lot of people this is easier said than done, particularly for younger engineers like myself. Younger engineers tend to have “more to prove” and are thereby far more emotionally invested in the code that they write, while older engineers whether by experience or simply by having written more code than their younger counterparts are able to distance themselves emotionally more easily from the code that they write. Not to say older engineers aren’t emotionally invested in their work, in my experience they typically are, it’s just a matter being better at picking battles.

Code review is a common sticking point for a lot of engineers, it’s incredibly important for both parties in a code review to judge the code objectively, if you are not, a code review can result in hurt feelings and resentment, personal differences bubbling up to the surface in a venue they don’t belong in. I think it’s immensely important to refer to code as an entity unto itself once a code review starts, phrases like “your code” are a major taboo. Separating the person who wrote the code from the code itself can help both the reviewer but also the original author of the code look at the changes in an objective light. “The code is overly complicated when all it should be doing is X.” “The patch doesn’t appropriately account for condition Y, which can happen if Z.” With a change in semantics, the conversation changes from one developer judging another’s work, to two developers objectively discussing whether or not the desired goal has been acheived with minimal downside. (Note: I’m presuming “proper code review” is being performed, devoid of nitpicking on minor style differences) You will find behavior like this in many successful open source projects that make heavy use of code review, the Git project comes to mind. When patches are posted to the mailing list, their merits are discussed as a separate entity, separated from the original author.

This same strategy of separating the individual from the code should also be applied to bugs in the code. When using git-blame(1) for example, there is a tendency to look at who authored the change, seek them out and pummel them with a herring. In a smaller team dynamic, as well as an open source environment, pinning “ownership” of a bug to a particular person is entirely non-constructive. Publicly citing and referencing somebody else’s mistake does nothing other than hurt that individual’s ego. The important part to refer to with git-blame(1) is the commit hash, and nothing else. With the conversation changed from “Jacob introduced a bug that causes X” into “Commit ff612a introduces a bug that causes X” those involved can then look at the code, and determine what about that code causes the issue. For simpler bugs the original author will typically pipe up with “Whoops, forgot about X, here’s a fix” but there are also cases where the original author didn’t know about the implications of the change, had no means of testing for X, or the bug was caused by another change the original author wasn’t privvy to. If the code is not separate from the individual, those latter cases can be tension points between developers that need not exist, making it all the more important (especially in small teams) to discuss changes openly and objectively.

With code decoupled from the author himself, how does the author maintain that same sense of pride and ownership? The original author should be charge with making any changes that arise out of a code review (naturally) but also should maintain responsibility for that portion of code moving forward; this added responsibility ensures less “fire and forget” changes and adds more pressure on the code reviews to yield improvements to the stability and readability of new code.

As soon as more than one developer is working on a project, it becomes increasingly important to recognize the difference between the “works of art” and the artist himself. The ceilings of the Sistine Chapel are an incredible piece of art, not because they were painted by Michelangelo. Writing code should be no different, the art is not the artist and vice versa.

Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I’m sure you’re thinking “But Dr. Tyler, Python is a dynamic language!”, it is indeed, but that does not mean there aren’t static variables.

The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you’re dealing with, it makes far more sense. Let’s take this static class variable for example:

>>> class Foo(object): ... my_list = [] ...

f = Foo() b = Foo()</code>

You’re trying to be clever, defining your class variables with their default variables outside of your __init__ function, understandable, unless you ever intend on mutating that variable. >>> f.my_list.append('O HAI')

print b.my_list [‘O HAI’] </code>

Still feeling clever? If that’s what you wanted, I bet you do, but if you wanted each class to have its own internal list you’ve inadvertantly introduced a bug where any and every time something mutates my_list, it will change for every single instance of Foo. The reason that this occurs is because my_list is tied to the class object Foo and not the instance of the Foo object (f or b). In effect f.__class__.my_list and b.__class__.my_list are the same object, in fact, the __class__ objects of both those instances is the same as well. >>> id(f.__class__) 7680112

id(b.class) 7680112</code>

When using default/optional parameters for methods you can also run afoul of statics in Python, for example:>>> def somefunc(data=[]): ... data.append(1) ... print ('data', data) ...

somefunc() (‘data’, [1]) somefunc() (‘data’, [1, 1]) somefunc() (‘data’, [1, 1, 1]) </code>

This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you’re adding the variable data to the somefunc.func_defaults tuple, which is being mutated when the function is being called. Bad programmer!

It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they’re difficult to spot, frustrating to deal with (“who the hell is changing my variable!”) and most importantly, easily prevented with a little understanding of how Python scoping works.

PYRAGE!

My “roots” in the open source community come from the BSD side of the open source spectrum, my first major introduction being involvement with FreeBSD and OpenBSD. It is not surprising that my licensing preferences fall on the BSD (2 or 3 clause) or MIT licenses, the MIT license reading as follows:<blockquote><p>Copyright (c) [year] [copyright holders]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.</blockquote> I bring the subject up because I wanted to address a brief "kerfuffle" that occurred recently on the Eventlet mailing list with the maintainer of gevent, a fork/rewrite of Eventlet. Both projects are MIT licensed which gives anybody that would like to fork the source code of either project a great deal of leeway to hack about with the code, commercialize it, etc. **Disclaimer**: I personally am a fan of Eventlet, use it quite often, and have recently taking up maintaining Spawning, a WSGI server that supports multiple processes/threads, non-blocking I/O and graceful code reloading, built on top of Eventlet. The "kerfuffle" occurred after Ryan, the maintainer of Eventlet, took a few good modules from gevent; *shocking* as it may seem, a developer working with liberally licensed code took liberally licensed code from a similar project. The issue that the maintainer of gevent took with the incorporation of his code was all about attribution: > I don't mind you borrowing code from gevent, the license permits it. However, please make it clear where are you getting the code from. Upon first reading the email, I doubled over to the [eventlet source on Bitbucket](http://bitbucket.org/which_linden/eventlet/), checked the files that were incorporated into the codebase ([timeout.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/timeout.py) and [queue.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/queue.py))and sure enough the copyright attributing the original author were still in tact, surely this is a non-issue? Unfortunately not, license pedantry is an open source community past-time, right up their with drinking and shouting. When I replied mentioning that the copyrights were correctly in place, mentioning that both projects were MIT licensed so both constraints of the license were met, that is, the MIT license notice was included with the code. In essence the disagreement revolves around what the phrase "this permission notice shall be included" entail, my interpretation of the license is such that the MIT license itself shall be included, not the specific file with additions from one project to another; after sending off my mail, I received the following reply: > Ok, it's acceptable to use one LICENSE file but only if the copyright notice from gevent is present unchanged. > > That is, take the notice from here http://bitbucket.org/denis/gevent/src/tip/LICENSE (the line with the url), and put it into eventlet's LICENSE, on a separate line. (It's OK to add "Copyright (c) 2009-2010" before it to make it in line with others). > > That would settle it. Slightly pedantic in my opinion, the MIT license enumerates a line for copyright holders which has been hijacked for other information that the maintainer of gevent would like to propagate, I don't necessarily agree, but this is a mailing list not a court of law, so I'll allow it. The thread continues: > The license did not change. I've only updated the copyright notice to include the url of the project to protect against abusive borrowers, that's it. This is where I draw the line, go all in, plant my flag in the sand and other unnecessary metaphors. **Abusive borrowers?** Analyzing the semantics of the phrase alone makes my head hurt, I have a mental image of two old ladies wherein one says to the other: "may I borrow a cup of sugar, you horse-faced hunch-backed bucket of moron?" The rest of the email is full of similarly head-hurting quotes, for brevity I won't include them here (you can read the thread [in the archives](https://lists.secondlife.com/pipermail/eventletdev/2010-February/000731.html)). I'm simply dumbfounded by the ignorance of what the MIT license actually *means*, unlike the LGPL or the GPL license which were specifically drafted to protect against "abusive borrowers", such as Cisco, the MIT license is so open it's *almost* public domain. To a certain extent I can understand the emotions behind the thread on the mailing list, I don't agree with them. If you're seeking attribution past the copyright line in a header, than perhaps the "original" [4 clause BSD license](http://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_.28original_.22BSD_License.22.29) is for you, or perhaps the LGPL or GPL which give you more control over what happens to the source code you originate? If this is what you're after, the MIT license is the wrong license.

A few weekends ago ET and I had some engagement photos taken, I’m told this is normal, by the husband-and-wife team from Tibidabo Photography, Bob and Becky. The duo met us at one of my favorite spots in San Francisco: Duboce Ave and Buena Vista Ave East after which we ran around in Buena Vista Park taking a few shots, then down to Baker Beach. As much as I hate having my picture taken, they did a wonderful job and grabbed some really stellar shots.

In my spurious free time I maintain a few Python modules (py-yajl, Cheetah, PyECC) and am semi-involved in a couple others (Django, Eventlet), only one of which properly supports Python 3. For the uninitiated, Python 3 is a backwards incompatible progression of the Python language and CPython implementation thereof, it’s represented significant challenges for the Python community insofar that supporting Python 2.xx, which is in wide deployment, and Python 3.xx simultaneously is difficult.

As it stands now my primary development environment is Python 2.6 on Linux/amd64, which means I get to take advantage of some of the nice things that were added to Python 3 and then back-ported to Python 2.6/2.7. Regular readers know about my undying love for Hudson, a Java-based continuous integration server, which I use to test and build all of the Python projects that I work on. While working this weekend I noticed that one of my C-based projects (py-yajl) was failing to link properly on Python 2.4 and 2.5. It might be easy to cut-off support for Python 2.4, which was first released over four years ago, there are still a number of heavy users of 2.4 (such as Slide), in fact it’s still the default /usr/bin/python on Red Hat Enterprise Linux 5. What makes this C-based module special, is that thanks to Travis, it runs properly on Python 3.1 as well. Since the Python C-API has been fairly stable through the 2 series into Python 3, maintaining a C-based module that supports multiple versions of Python.

In this case, it’s as easy as some simple pre-processor definitions:#if PY_MAJOR_VERSION >= 3 #define IS_PYTHON3 #endifWhich I can use further down the line to modify the handling some of the minor internal changes for Python 3:#ifdef IS_PYTHON3 result = _internal_decode((_YajlDecoder *)decoder, PyBytes_AsString(bufferstring), PyBytes_Size(bufferstring)); Py_XDECREF(bufferstring); #else result = _internal_decode((_YajlDecoder *)decoder, PyString_AsString(buffer), PyString_Size(buffer)); #endif

Not particularly pretty but it gets the job done, supporting all major versions of Python.

Python on Python

Writing modules in C is fun, can give you pretty good performance, but is not something you would want to do with a large package like Django (for example). Python is the language we all know and love to work with, a much more pleasant language to work with than C. If you build packages in pure Python, those packages have a much better chance running on top of IronPython or Jython, and the entire Python ecosystem is better for it.

A few weeks ago when I started to look deeper into the possibility of Cheetah support for Python 3, I found a process riddled with faults. First a disclaimer, Cheetah is almost ten years old; it’s one of the oldest Python projects I can think of that’s still chugging along. This translates into some very old looking code, most people who are new to the language aren’t familiar with some of the ways the language has changed in the past five years, let alone ten.

The current means of supporting Python 3 with pure Python packages is as follows:

1. Refactor the code enough such that 2to3 can process it
2. Run 2to3 over the codebase, with the -w option to literally write the changes to the files
3. Test your code on Python 3 (if it fails, go back to step 1)
4. Create a source tarball, post to PyPI, continue developing in Python 2.xx

I’m hoping you spotted the same problem with this model that I did, due to the reliance on 2to3 you are now trapped into always developing Python targeting Python 2. This model will never succeed in moving people to Python 3, regardless of what amazing improvements it contains (such as the Unladen Swallow work) because you cannot develop on a day-to-day basis with Python 3, it’s a magic conversion tool away.

Unlike with a C module for Python, I cannot #ifdef certain segments of code in and out, which forces me to constantly use 2to3 or fork my code and maintain two separate branches of my project, duplicating the work for every change. With Python 2 sticking around on the scene for years to come (I don;t believe 2.7 will be the last release) I cannot imagine either of these workflows making sense long term.

At a fundamental level, supporting Python 3 does not make sense for anybody developing modules, particularly open source ones. Despite Python 3 being “the future”, it is currently impossible to develop using Python 3, maintaining support for Python 2, which all of us have to do. With enterprise operating systems like Red Hat or SuSE only now starting to get on board with Python 2.5 and Python 2.6, you can be certain that we’re more than five years away from seeing Python 3 installed by default on any production machines.