Welcome to my blog where I write about software
development, cycling, and other random nonsense. This is not
the only place I write, you can find more words I typed on the Buoyant Data blog, Scribd tech blog, and GitHub.
While working at Slide I had a tendency to self-assign major projects,
not content with things being “good-enough” I tended to push and over-extend
myself to improve the state of Slide Engineering. Sometimes these projects
would fail and I would get uncomfortably close to burning myself out, other times,
such as the migration from Subversion to Git, turned out to be incredibly rewarding
and netted noticable improvements in our workflow as a company.
One of my very first major projects was upgrading our installation of Cheetah from 1.0 to
2.0, at the time I vigorously hated Cheetah. My distain of the templating system stemmed
from using a three year old version (that sucked to begin with) and our usage of Cheetah which
bordered between “hackish” and “vomitable.” At this point in Slide’s history, the growth of
the Facebook applications meant there was going to be far less focus on the Slide.com codebase
which is where some of the more egregious Cheetah code lived; worth noting that I never “officially”
worked on the Slide.com codebase. When I successfully convinced Jeremiah and KB that it was worth
my time and some of their time to upgrade to Cheetah 2.0 which offered a number of improvements
that we could make use of, I still held some pretty vigorous hatred towards Cheetah. My attitude was
simple though, temporary pain on my part would alleviate pain inflicted on the rest of the engineering
team further down the line. Thanks to fantastic QA by Ruben and Sunil, the Cheetah upgrade went down
relatively issue free, things were looking fine in production and everybody went back to their
regularly scheduled work.
Months went by without me thinking of Cheetah too much until late 2008, Slide continued to write
front-end code using Cheetah and developers continued to grumble about it. Frustrated by the
lack of development on the project, I did the unthinkable, I started fixing it. Over the Christmas
break, I used git-cvsimport(1) to create a git repository from the Cheetah CVS repo hosted with
SourceForge and I started applying patches that had circulated on the mailing list. By mid-March
I had a number of changes and improvements in my fork of Cheetah and I released “Community
Cheetah”. Without project administrator privileges on SourceForge, I didn’t have much of a choice
but to publish a fork on GitHub. Eventually I was able to get a hold of Tavis Rudd, the original
author of Cheetah who had no problem allowing me to become the maintainer of Cheetah proper,
in a matter of months I had gone from hating Cheetah to fulfilling the oft touted saying “it’s
open source, fix it!” What was I thinking.
Thanks in part to git and GitHub’s collaborative/distributed development model patches started to
come in and the Cheetah community for all intents and purposes “woke up.” Over the course of the
past year, Cheetah has seen an amazing number of improvements, bugfixes and releases. Cheetah now
properly supports unicode throughout the system, supports @staticmethod and @classmethod decorators,
supports use with Django and now supports Windows as a “first-class citizen”. While I committed
the majority of the fixes to Cheetah, five other developers contributed fixes:
In 2008, Cheetah saw 7 commits and 0 releases, while 2009 brought 342 commits and 10 releases;
something I’m particularly proud of. Unforunately since I’ve left Slide, I no longer use Cheetah
in a professional context but I still find it tremendously useful for some of my personal projects.
I am looking forward to what 2010 will bring for the Cheetah project, which started in mid-2001 and has
seen continued development since thanks to a number of contributors over the years.
Earlier while talking to Ryan I decided I’d try to coin the term “pyrage” referring to some frustrations I was having with some Python packages. The notion of “pyrage” can extend to anything from a constant irritation to a pure “WTF were you thinking!” kind of moment.
Not one to pass up a good opportunity to bitch publicly, I’ll elaborate on some of my favorite sources of “pyrage”, starting with generic exceptions. While at Slide, one of the better practices I picked up from Dave was the use of specifically typed exceptions to specific errors. In effect:
class Connection(object):
## Pretend this object has "stuff"
pass
class InvalidConnectionError(Exception):
pass
class ConnectionConfigurationError(Exception):
pass
def configureConnection(conn):
if not isinstance(conn, Connection):
raise InvalidConnectionError('configureConnection requires a Connection object')
if conn.connected:
raise ConnectionConfigurationError('Connection (%s) is already connected' % conn)
## etc </code>
Django, for example, is pretty stacked with generic exceptions, using builtin exceptions like ValueError and AttributeError for a myriad of different kinds of exceptions. urllib2’s HTTPError is good example as well, overloading a large number
of HTTP errors into one exception leaving a developer to catch them all, and check the code, a la:
try:
urllib2.urlopen('http://some/url')
except urllib2.HTTPError, e:
if e.code == 503:
## Handle 503's special
pass
else:
raise
A weekend or two ago I sat down and created an Amazon wishlist of stuff that I would like to purchase. In the past I’ve found Amazon wishlists an ideal way of saying thanks to a number of folks in the open source community whose work I value, at one point I didn’t desire any new gadgets and went through a large number of gift cards buying goodies for some of my favorite hackers on Amazon.
A few weeks ago when I started working more on py-yajl I discovered that there are actually a number of Python developers who work(ed) with JSON parsers in the Bay Area. As luck would have it Lloyd, the author of Yajl, is going to be in town next weekend for Add-on-Con, time to meet up and have some beers! I’ve invited the authors of: simplejson, jsonlib, jsonlib2 and a few other Python hackers in the Bay Area that I know of.
If you’re in San Francisco this Saturday (Dec. 12th) and loves you some Python, don’t hesitate to swing by 21st Amendment around 1pm-ish to join us!
Update: Some of this information is out of date. Instead of pushing to the
gerrit master branch I recommend setting up
“replication” and using the
“Submit” button inside of the “Review” page.
While working at Slide, I became enamored with the concept of cooperative threads (coroutines) and the in-house library built around greenlet to implement coroutines for Python. As an engineer on the “server team” I had the joy of working in a coro-environment on a daily basis but now that I’m “out” I’ve had to find an alternative library to give me coroutines: eventlet. Interestingly enough, eventlet shares common ancestry with Slide’s internal coroutine implementation like two different species separated thousands of years ago by continental drift (a story for another day).
A few weekends ago, I had a coroutine itch to scratch one afternoon: an eventlet-based image server for applying transforms/filters/etc. After playing around for a couple hours “PILServ” started to come together. One of the key features I wanted to have in my little image server project was the ability to not only pass the server a URL of an image instead of a local path but also to “chain” transforms in a jQuery-esque style. Using segments of the URL as arguments, a user can arbitrarily chain arguments into PILServ, i.e.:
http://localhost:8080/flip/filter(blur)/rotate(45)/resize(64x64)/<url to an image>
At the end of the evening I spent on PILServ, I had something going that likely shows off more of the skills of PIL rather than eventlet itself but I still think it’s neat. Below is a sample of some images transformed by PILServ running locally:
Perhaps the title is a bit too much ego stroking, yes, I did write the fastest Python module for decoding JSON strings and encoding Python objects to JSON. I didn’t however write the parser behind the scenes.
Over the summer I discovered “Yet Another JSON Library” on GitHub, written by Lloyd Hilaiel, jonesing for a Saturday afternoon project I started the “py-yajl” project to see if I could implement a Python C module atop Lloyd’s marvelous parsing library. After tinkering with the project for a while I got a working prototype building (learning how to define custom types in Python along the way) and let the project stagnate as my weekend ended and the workweek resumed.
A little over a week ago “autodata”, another GitHub user, sent me a “Pull Request” with some minor changes to make py-yajl build cleaner on amd64; my interest in the project was suddenly reignited, amazing what a little interest can do for motivation. Over the 10 days following autodata’s pull request I discovered that a former colleague of mine and fellow GitHub user “teepark” had forked the project as well, working on Python 3 support. Going from zero to two people interested in the project, I quickly converted the code from a stagnant, borderline embarrassing, dump of C code into a leak-free, swift JSON library
for Python. Not one to miss out on the fun, I pinged Lloyd who quickly became as enamored with making py-yajl the best Python JSON module available, he forked the project and almost immediately sent a number of pull requests my way with further optimizations to py-yajl such as:
Swapping out the use of Python lists to a custom pointer stack for maintaining internal state
Accelerating parsing and handling of Number objects
Pruning a few memory leaks here and there
Thanks to mikeal’s JSON post and jsonperf.py script, Lloyd and I could both see how py-yajl was stacking up against cjson, jsonlib, jsonlib2 and simplejson; things got competitive. Below are the most recent jsonperf.py results with py-yajl v0.1.1:
Over the coming days or weeks (as time permits) I’m planning on adding JSON stream parsing support, i.e. parsing a stream of data as it’s coming in off a socket or file object, as well as a few other miscellaneous tasks.
Given the nature of GitHub’s social coding dynamic, py-yajl got off the ground as a project but Yajl itself gained an IRC channel (#yajl on Freenode) and a mailing list (yajl@librelist.com). To date I have over 20 unique repositories on GitHub (i.e. authored by me) but the experience around Yajl has been the most exciting and finally proved the “social coding” concept beneficial to me.
In addition to RSS feeds, one of my favorite sources of reading material is the Git mailing list; I’m not really active, I simply enjoy reading the discussions around code and the best solutions for certain problems. If you read the list long enough, you’ll start to appreciate the time and attention the Git core developers (spearce, peff and junio (a.k.a. gitster)) put into cultivating the code and in cultivating new contributors. Of all the open source projects I watch to one extent or another, Git is very effective at bringing in new contributors and getting their contributions vetted for inclusion.
If you’re a heavy Git user (like me) you can certainly see the results of their tireless efforts, Junio’s (git.git’s maintainer) in particular. I highly recommend checking out his Amazon wishlist to thank him for his efforts.
My journey at Slide comes to an end today, when I leave this evening
I will once again return to being a free agent (if only for two days).
Some of my coworkers have casually referred to my writings over the
past few days of my “memoirs”, which isn’t too far off to be honest.
When I leave Slide this evening, my employment will have accounted
for roughly 10% of my entire life and 50% of my adult life. Writing
my side of the story down, to some extent, has been more about telling
a story to myself and less about telling it to anybody else (apologies).
So much of my time spent at Slide has been in a state of controlled
choas that at times it’s hard for me to remember when things were done,
who was doing them and the order in which they happened.
The two questions I’ve invariably gotten since I gave my notice and
subsequently started writing this series of posts have been:
why are you leaving and where are you going? My reasons for leaving
are irrelevant, I will say that if I could take the people I’ve been working
with at Slide with me, I would. I’ve learned such an incredible amount
from those that I’ve worked with, both technical and non-technical, while
at Slide. Whenever I would pitch friends on the idea of joining Slide, my
take-home point was always “when you join Slide, you will not be the
smartest person there.” I feel lucky that I was given the opportunity to
“come of age” as a young engineer in a company of so many tremendously talented
individials, given the chance for a do-over, I would still play my cards the
way I’ve played them. I joined Slide a punk kid from Texas, I’m leaving
Slide a slightly-more-learned punk kid from Texas.
As to where I am going, after an extended vacation of Saturday and Sunday,
I will be joining my second startup ever (Slide’s a pretty good
first time out) on Monday. When I started looking at other companies I had a couple
of criteria, I wanted to join a smaller team (Slide’s upwards of a hundred people
or so) and I had to really like who I would be working with. Tristan, Jesse, Can (John)
and the team from Apture fit both criteria. I’m not going to go into detail
about what I’m really going to be doing there or where Apture is going as a
company. I will say that after two and a half years of working and studying
at Slide, I’m looking forward to employing what I’ve learned and continuing
my education at Apture.
Prior to joining Slide, a friend of mine “whurley” had nicknamed
me the “Angry Young Man” which I promptly put on my first set of
business cards (my current business cards list my title as “Meta-Chief
Platform Architect, Enterprise Edition”, I received them after mentioning a failed
poaching attempt by LinkedIn to Max); when Top Friends went dark
on Facebook, I was a little more than an “angry young man.”
Given my close involvement with the product, the amount of sleepless
nights working on it, the actions against Top Friends felt personal to
me, regardless of the posturing between Slide and Facebook’s
executives. As hours turned into days offline, it became clear to me
that the suspension of the application was far less about our privacy hole
and far more about Facebook making an example out of Top Friends to
the rest of the platform development community. The message was heard loud
and clear by the majority of the developers that I knew, this is not your
platform, these are not your users and you will play by our rules or we
will wipe you from the face of the site. Building on the platform
was not only no more fun, it was also a risky business decision.
At the time of the suspension, Keith and I had already started discussing
what a “TopFriends.com” might look like, as the signals of platform
instability for applications were already being sent. When Top Friends went
offline, I prepared a few page outline for Max and Keith detailing “my vision”
for what Top Friends would become, I was convinced by that time that its
future lie as a social network unto itself, rather than a network contained
bu another network (yo dawg..). Not content to simply be “vanity and personal
expression” inside of Facebook, I wanted Top Friends to become a separate
entity by itself, your VIP club on the internet, at one point there was
even executive support for the drawing of users away into a destination site
for Top Friends. When the seven days of suspension were over and Top Friends
came back online, Slide’s strategy shifted drastically. Our new mission for TF on
Facebook was to “get as close to Facebook as you can,” we were to integrate
into a user’s experience as much as conceivably possible. Previously we
had wanted to run as far away from Facebook as we could, taking our users
with us, but the fear that was enstilled by the application suspension
caused us to rethink that stance and push Top Friends to be a squeeky clean
platform citizen, while we contemplated a possible exodus for FunSpace and SuperPoke!.
Around this time in Slide’s history I became quite jaded and cynical with
regards to the platform, Top Friends had been neutered by Facebook, and my
notion of what Top Friends should have been was neutered by Max. Regardless,
we still had plenty of work that needed to be done to try to succeed with our
new strategy. Months prior, Tony Hsieh (not the Zappos guy) the original
Top Friends PM had failed to win the visa lottery and moved back to China,
leaving TF without a product manager for some time. While we continued to
look for senior PM to take on the role, I had to play both product and engineering
manager (with help in both places every now and again). Quite the twist of
fate for me, I had often poked fun at PMs at Slide, once creating a powerpoint (one should
speak the language) titled “PM Flowchart”. The presentation consisted of one slide,
with a fairly simple state diagram on it, one block labeled “Write Spec” had an
arrow pointing to another block labeled “Bitch.” which pointed back at “Write Spec”.
Suffice to say, product managers and I usually had a tenuous relationship.
Passionate about the product to begin with, I started meeting more and more often
with Max and Keith to discuss product strategy for TF, in between doing my “real job” of
Engineering Manager. Some meetings Keith and Max would square off and I would sit back
and watch, other times Keith and I squared off against Max, I rarely took Max’s side against
Keith’s though. Not that I always disagreed with Max, but he was at a slight disadvantage
in these discussions, Keith and I generally shared a lot of fundamental ideas of what TF should be,
stemming from months of discussing the product by his desk before he ever “officially” worked
with the project. The transition over a year and a half from quivering in fear as the director
of engineering cursed at me on Dave’s house phone, to arguing with the CEO about
the product he pitched me on, was surreal to say the least. How I didn’t get fired
is either a testament to my charm or Max’s patience.
In fall of 2008, when Seema finally joined as the Top Friends product manager, not only was
I more than ready to relinquish the post, Top Friends was in the midst of
an identity crisis. Our “facebook zerg rush” strategy of getting closer and closer
to the platform played out as you might of expected (hindsight and all), Facebook
redesigned the profile, changed viral communications channels and did a lot
of things that were likely good for Facebook, but terrible for applications.
TF had a lot of momentum on the “old profile” thanks to users dragging the TF
profile box all the way up on their profiles. When Facebook rolled out their new profile
which put applications not in the backseat, but in the way-back seat, the strategy
of “be lovey dovey with Facebook” started to break down, they weren’t being
lovey dovey back.
Times were also changing outside of Top Friends at Slide, the SuperPoke! Pets product was
starting to take off and actually make money directly from users. This was
important! Users, giving us money, for pixels! Brilliant! Being a much more reliable
revenue stream than the advertising oriented model that FunSpace, SuperPoke! and Top Friends
had been built around, Pets quickly became the “top” product at Slide.
With ad revenue drying up for Top Friends, we were tasked
with experimenting with virtual currency (like Pets) and ultimately “premium items”
(like Pets) within Top Friends. It seemed almost as if Top Friends was changing
visions, strategies and directions on a bi-weekly basis. One week we were building
virtual currency experiments with “Top Dollars”, the next, virtual economy
experiments with an “Own your friends’ profiles” feature, the next, premium virtual
goods with “Top Gifts”. As the “Top Friends guy” and the manager of the engineering team,
I was so confused and disoriented about what we actually did and where we were
actually heading, I didn’t stand a chance at convincing Paul, Geoff and Jason of it.
2008 winding down, the writing was on the wall, Top Friends was not going to live long,
at least the Top Friends Team wasn’t. We had gained a reputation of being very
self-sufficient and competent, but with that autonomy came uncertainty from outsiders.
I regularly had to remind coworkers that I was a Slide engineer, not a Top Friends
engineer, regardless of the TF team’s internal view of itself as a “microstartup.”
When we failed to meet goals set out for us, it was decided that the staff behind
Top Friends were too valuable to spend time on a failing product.
Jason, Paul and Seema went to start a new project,
while Geoff and I, together since the desktop client days, joined the Server/Infrastructure team.
My personal “love” for Top Friends had all but dissolved by this point, I was sick of
Top Friends, I was sick of Facebook, I was sick of policy, I didn’t care all that
much about the product anymore. The breaking up of the team though, was crushing.
As far war metaphors go, the TF team was a small rag-tag group of guerrillas, capable
of taking large projects and finishing them in record time. We often talked about
what we did as “playing jazz music” because our work had an improvisational style,
but the trust and understanding of where we all fit into the act, allowed us
to tackle large tasks in stride; that was all over though. The dream team was broken up.
My time on the server team at Slide is unfortunately a boring story of working
with stellar engineers capable of writing solid code and deploying it without
incident. As exciting as wood filler “this worked out just fine, the end.”
After years of frenzy with Top Friends and the Facebook platform,
my first project for the server team took three weeks to build, was pushed
without a hitch and has only required two minor updates since. With my nose
to the grindstone building services and scalable architecture, I went
months without particularly concerning myself with “product direction”, company
strategy and their ilk. The closest I would come to application development would be
jumping up into application code to fix bugs, all the while cursing app developers’
laziness while conveniently forgetting how often I was guilty of the same offense
in my tenure with Top Friends.
When I finally stuck my head back up, near the end of the summer, I started to
realize that I was working at a different company than I remember joining. Slide
had grown tremendously and changed direction once again. Since stepping back from
the front-lines, I had changed and Slide had changed too.
It was about time Slide and I started seeing other people.
When I finished up writing part 1 of my journey at Slide yesterday, I had
just recounted becoming “the Top Friends guy”, savvy readers might have noted
that I had not moved off of Dave’s couch at the time. I am uncertain whether
it is a record to be proud of, but I held the position of “the guy on Dave’s
couch” for two months. With the leadup to the “F8” conference I didn’t have a
whole lot of time to find an apartment, Dave being an all around nice guy and
amazing cook, wasn’t helping my motivation to leave either. That said, I’m a
delightful house guest, honest.
Shortly after the initial successes of the Top Eight product, and the launch of
“FunWall” (renamed “FunSpace” later), Slide quickly converted the desktop client
team to the “Facebook Team” with 4-5 engineers hacking on Facebook applications
to capitalize as quickly as possible on the wild-west nature of the platform at
the time. We subsequently launched another couple apps, such as “My Questions” an application
that allowed you to poll your friends (likely our most “useful” application). I
ended up writing another application alongside Top Eight called “Fortune Cookie”,
contrasted to My Questions, it was probably our most useless application. The application
was absolutely brilliant (Mike and Max get credit here again), the profile box for the application was a picture of a fortune
cookie with a fortune overlaid. Brilliant. If/when the user clicked through to the
application’s canvas page, they were met with a simple grid of checkboxes and friends’
faces, checkboxes checked with a giant blue button that said “Invite your Friends!”.
Never underestimate the power of “Select All”, Fortune Cookie exploded, alongside
our “Magic 8 Ball” application (guess what that was), it spread through the Facebook
ecosystem like an epidemic. By mid June Top Eight was renamed Top Friends after
we bumped the number of “top friends” you could list from 8 to 24 (innovation!);
with the power of an intrisicly simple value-proposition to users, 24 friend tiles
and “Select All”, Top Friends held the rank of #1 application on Facebook. Following
Top Friends was iLike, a major initial success, with Fortune Cookie pulling in third
place. Further down the list were a couple of familiar applications: Free Gifts, created
by Zach Allia, a Northwestern student at the time and a regular on the #facebook channel
on Freenode; Rock You!’s “X Me” application was likely one of the first acquisitions on the Facebook platform, after
being created by a student who joined the #facebook channel frantically
asking for help as his server was crumbling under the load of pure virality, and SuperPoke!
an application created by a then part-time Microsoft employee and two friends.
The first couple weeks of the Facebook platform were sheer insanity, determined to one-up our
competitors Rock You!, Slide acquired SuperPoke! and the three engineers that wrote it,
Nik, Will and Jonathan. Slide was determined to own the market of “virtually do virtual things to your virtual friends
on Facebook”. In short order the SuperPoke team moved down from Seattle to join the “Facebook Team”
in Slide’s office at 2nd and Howard, Jon went to the metrics team (being a PhD and all) while
Nik and Will shared a desk and started learning Python to port SuperPoke! over to Slide’s stack
to allow it to scale faster and better than could have been possible on the PHP/MySQL stack it
used at the time. Prior to joining Slide, the SuperPoke! application icon was some picture of a goat
Nik had plucked from the internets, by joining Slide they had access to real
designers, not goats from Google Image Search. Slide’s most senior designer, Johnnie, can
be credited with helping define the brand that would ultimately be synonymous with the Facebook
platform and Slide: the SuperPoke sheep. While SuperPoke! and X Me battled it out for 4th and 5th place
in the application rankings, journalists started writing articles discussing the Facebook platform,
in both positive and negative light, without fail mentiong the absurdity of “throwing sheep” at
your friends. I always got the impression that Mark Zuckerberg would have considered the
Facebook platform successful when IBM ported Lotus Notes to it, being a “utility fetishist”, I
can only imagine how “delighted” he must have been with the top applications on the platform
being the likes of Top Friends, Fortune Cookie, Horoscopes, Graffiti, X Me and SuperPoke!.
After the SP guys had joined Slide, Facebook hosted a mid-morning event at their Palo Alto office
to help kickstart some developer relations and have top application developers do some
lightning-round style presentations. The meeting starting at 9, it was only logical that
Nik, Will, Max and I meet at the Slide offices at 8:45; we piled into Max’s BMW M3 (a gorgeous car,
I highly recommend it) and sped southwards from San Francisco on the 101. Despite driving between 90-100mph
through rush-hour traffic towards Palo Alto, we arrived fashionably late; walking in
during a presentation, Dave McClure announced to the whole room “Slide has arrived.”
Roll around in an M3 enough, have people announce your arrival enough and you too will feel like
a Web 2.0 rockstar. Being the “Top Friends guy”, I certainly had a bit of an ego going, I still kind
of do, but I’m far more modest now about being a complete badass.
The summer of 2007 was mostly a blur, the majority of my “workdays” ended up being 14-16 hours usually
ending with Geoff, Sergio, Kasey and I drinking into the wee hours of the morning, pushing code and
smoking on the fire escape (building management didn’t really care for that part). The night before
the iPhone launched, a bunch of Sliders had arranged to wait in line in shifts at Apple’s Market St store
(we were third in line). Given my schedule at the time, I worked most of the night and then manned
the 4-7 a.m. shift in line. I didn’t even want an iPhone but Tony, the product manager we hired
for Top Friends, and I hung out on the sidewalk, smoked fancy cigars and watched the streets get
cleaned. My (now) fianceé was still in Texas finishing up with school, so I had nothing to
do but hang out, drink, smoke, write code, push the site and sleep every now and again. My
apartment, right in the middle of the colorful Tenderloin district, only served as a place to
shower and crash. For the duration of my lease, I didn’t own any dishes and rarely had
anything in the fridge other than left-over pizza and Cokes.
By the latter part of 2007 we hired Keith Rabois to be the VP of Business Development, presumably to help us ink
deals with big important companies about big important things (with big important sacks of money).
Initially, I hadn’t a clue what the hell Keith did, other than walk around in his shiney silk
shirts talking on his fancy iPhone, loud enough to hear across the office. The layout of Slide’s
office was such that on one end was the open floorplan engineering “pit” and on the other end,
separated by a ping pong table and a copy machine were the “non-engineers”. The ping pong
table was usually as far as I went. At some point, I don’t remember exactly when, I started
consulting with Keith on product related matters. He had this chair by his desk, so I would
stroll over, plop down and gab for longer than he probably had time for about subjects ranging
from the latest Facebook gossip to long-term strategy; Keith’s involvement with
Top Friends would only increase from then moving forward.
By the beginning of 2008, the Facebook platform wasn’t fun anymore. Too many emails contained
the words “policy” and “violation” and often dastardly combinations of the two. At the same time,
Slide had upped its commitment to Top Friends hiring Jason, who I had known for some time
from the #facebook IRC channel, his compatriot Paul, and assigning Geoff, a senior QA engineer
who had put up with my shit on the client team since I joined the company months earlier.
I was promoted to Engineering Lead and shortly thereafter to Engineering Manager. My role had changed dramatically, no longer
simply just a monkey coding like there was no tomorrow, I now had people I had to be accountable
to, all the miserable hacks I had thrown into Top Friends in the previous 8 months I had to
sheepishly explain to Jason and Paul, mentioning from time to time how I could
do it better given the time.
Jason and Paul being hired and assigned to my team was likely the luckiest thing that ever happened to me at Slide,
overnight I went from a hard-working “army of one” to part of a team of four hard-working
bone crushers with an incredible drive to succeed. In a few short months we had shipped an “Awards”
feature, built out a “Top Friends Profile” and started pushing our way back to the top.
In June, a reporter for CNet reported on a hole in the Top Friends Profile that allowed
a user to view information about other users they could not have otherwise seen. The
reporter used this an instrumental piece of a larger article bashing Facebook
on their privacy record and the openness of the Faceobok platform.
When Keith texted me that night, I rushed home and pushed a fix for the hole within the
hour, went to dinner by myself and had the worst Pad Thai I’ve ever eaten, watching
the exchange of emails between Slide’s and Facebook’s executive team on my Blackberry.
As some of you may, or may not know, this friday October 23rd will be my final
day as a Slide employee. With my journey at Slide nearing its completion, I wanted
to document some of how I’ve gotten here and where I’ve started, if for nobody other
than myself.
Officially I started at Slide April 2nd, 2007, though my journey to Slide started
far earlier. At the end of my fourth semester at Texas A&M my then girlfriend, now fianceé
and I decided we were through with College Station and to move to San Antonio; most Texans would consider this a lateral move at best.
I had every intention of resuming my studies at UTSA following a brief stint at San Antonio College clearing up pre-requisites with a slightly lower price tag. By
the end of fall semester it had become clear that I wasn’t cut out for college, I
stopped attending and focused full time on software. At the time most of my experience
and contacts were through the Mac development community, primarily via IRC on the Freenode
network and developer mailing lists for various open source projects. Through
my involvement in the Bonjour mailing lists and work with the API, I had at one
point impressed Bonjour’s original inventor Stuart Cheshire enough to land
an interview at Apple for the Core OS group, working on Bonjour.
Sitting on the Continental flight out to San Jose, I practiced writing network services
using BSD sockets and pouring over as much C as I could possibly manage, all told I
likely wrote around 3 multicast service/client pairs on that flight. What I wasn’t prepared for
was the “computer science” nature of the interview; I bombed it with my rudimentary algorithms
knowledge and lack of experience working with C on a day to day basis. Fortunately, my last
interviewer of the day was Ernie Prabhakar then a product or marketing manager for the Core OS
group; Ernie indulged me in a very interesting conversation about Apple’s position in the open
source universe, product direction, etc. Despite bombing the technical portions of the interview,
I suppose Ernie saw enough enthusiasm in me to refer me to Dave Morin at Facebook (the two worked
together at some point).
Those that know Dave Morin understand that the man wields a Jobsian reality distortion field,
even via email 1500 miles away in Texas I felt the power of the field and was drawn to Facebook.
While I was ultimately disappointed to not have landed my then-dream-job at Apple, I was incredibly excited to be flying back to Silicon Valley to interview at Facebook. When I mentioned to daver (Dave Young) on IRC that I would be flying back out to
see the nice folks at Facebook in Palo Alto, he also arranged an interview at Slide the day after.
A number of factors likely lead to my failure to excite my interviewers at Facebook, not having a
Facebook account for one didn’t help, I also think I uttered “fuck” under my breath once or twice while
sketching out problems on a whiteboard. Considering my interview was done mostly in “the game
room” due to a scheduling error, I didn’t think I was being too unprofessional. As one could assume, my interview with Slide went substantially better, I accepted an offer to join Slide as
a junior software engineer working on their now defunct desktop application(s).
My start date was set for April 2nd, within three weeks I had terminated the lease on my apartment
in San Antonio, tossed, sold or otherwise gave away the majority of my belongings and furniture and packed my VW Jetta
to the brim and drove west. I didn’t particularly have a plan other than “show up, get to work”
(I was 21, how young and foolish), so I crashed on daver’s couch while I settled in and started
searching for an apartment.
My early days at Slide were all about getting up to speed on Python (Slide’s language of choice) and
ActionScript 2 (Slide’s only option for Flash at the time); I started helping with the Windows
client, mostly in the spagetti-driven Flash-based screensaver product. Towards the beginning of May,
Jeremiah (then Director of Engineering, now CTO) and Bobby (another engineer) were working with
some preview APIs from what would ultimately become the Facebook platform. For whatever reason I
started working on trying to incorporate some data from Facebook into our desktop client (optimal
synergy, etc) and became the third engineer working at Slide on the Facebook platform in its
infancy. As May came to completion, we (Slide) were invited to “F8” to unveil some of the
applications for their new platform we had built.
Donning my trusty brown cordoruy sport jacket, dark blue Slide t-shirt, I helped man Slide’s booth
presenting some of our apps: SlideShows and YouTube Skins (both products turned out to be utter
failures). As the business/presentation portion part of F8 wound down, I grabbed a 19” monitor
and told Max and Jeremiah that I wanted to stay for the hackathon but didn’t have any idea what to
hack on (being a desktop developer and all). Max leaned in and muttered “Top Friends” on his way
out, leaving me to set up shop with the only dual screen setup in the hackathon, at a lonely table by myself (I hadn’t
figured out how to socialize at that point). Coming from a desktop background, I hadn’t a clue
what I was doing, I could barely figure out how to get pages working on Slide’s infrastructure, let
alone all this FBML, FQL and JavaScript malarkey.
Fortunately in the days following the hackathon, I was able to enlist the help of Sergio, the best web front-end engineer
Slide had to offer to help me create a grid of drag and droppable images along with
some other pieces of front-end to make the application palatable. All said and done, if I remember correctly, Top Eight launched less than a
week after the platform did, my first “big” project at Slide. Originally I couldn’t get database
resources for the app, so I stashed the friends list inside of the profile FBML and then would
subsequently retrieve back from Facebook when I needed it, using regular expressions (had help
with that too) to pull the list of Facebook user IDs out; that hacked up solution lasted for all
of maybe 30 minutes on live as soon as everyone saw how god-awful slow it was.
Day three of Top Eight, I learned what “viral meant”. My parents had neglected to pay their phone
bill, taking my “family plan” number out with it, meaning I couldn’t receive the frantic calls
from Jeremiah as I slept-in that morning. Turns out by giving the Top Eight a callback URL with
Facebook that hit “www.slide.com” was proving impossible to load balance, resulting in a couple
hours of site issues for the rest of Slide, as Top Eight skyrocketed hundreds of thousands of users in a single day.
I awoke that morning to pounding on Dave’s door (I was still on their couch), opening it I
saw Carey (another desktop developer at Slide) who said “your phone’s off.” Not my preferred
way to wake up, but it sufficed. I sheepishly called Jeremiah on Dave’s house phone.
Jeremiah was pissed. Not “who ate the rest of my hummus”
pissed, righteously pissed, at me. Here I was, living on a friend’s (who I met on the internets)
couch, without a proper mailing address trying to figure out how this startup thing worked, and
Jeremiah was furious with me. If there were such a thing as an “ideal time” for an earthquake, I
would have gladly accepted that as an alternative.
Once the smoke cleared and tempers cooled, we looked at some of the installation and growth numbers of Top Eight
during the previous 6 hours; I had found myself a new job at Slide. From that day forth, I was
“the Top Friends guy.”
Last week @admc, despite being a big proponent of Windmill, needed to use WatiN for a change. WatiN has the distinct capability of being able to work with Internet Explorer’s HTTPS support as well as frames, a requirement for the task at hand. As adorable as it was to watch @admc, a child of the dynamic language revolution, struggle with writing in C# with Visual Studio and the daunting “Windows development stack,” the prospect of a language shift at Slide towards C# on Windows is almost laughable. Since Slide is a Python shop, IronPython became the obvious choice.
Out of an hour or so of “extreme programming” which mostly entailed Adam watching as I wrote IronPython in his Windows VM, IronWatin was born. IronWatin itself is a very simple test runner that hooks into Python’s “unittest” for creating integration tests with WatiN in a familiar environment.
I intended IronWatin to be as easy as possible for “native Python” developers, by abstracting out updates to sys.path to include the Python standard lib (adds the standard locations for Python 2.5/2.6 on Windows) as well as adding WatiN.Core.dll via clr.AddReference() so developers can simply import IronWatin; import WatiN.Core and they’re ready to start writing integration tests. When using IronWatin, you create test classes that subclass from IronWatin.BrowserTest which takes care of setting up a browser (WatiN.Core.IE/WatiN.Core.FireFox) instance to a specified URL, this leaves your runTest() method to actually execute the core of your test case.
Another “feature”/design choice with IronWatin, was to implement a main() method specifically for running the tests on a per-file basis (similar to unittest.main()). This main method allows for passing in an optparse.OptionParser instance to add arguments to the script such as “–server” which are passed into your test classes themselves and exposed as “self.server” (for example). Which leaves you with a fairly straight-forward framework with which to start writing tests for the browser itself:
#!/usr/bin/env ipy
The import of IronWatin will add a reference to WatiN.Core.dll
and update sys.path to include C:\Python25\Lib and C:\Python26\Lib
so you can import from the Python standard library
import IronWatin
import WatiN.Core as Watin
import optparse
class OptionTest(IronWatin.BrowserTest):
url = ‘http://www.github.com’
def runTest(self):
# Run some Watin commands
assert self.testval
if name == ‘main’:
opts = optparse.OptionParser()
opts.add_option(‘–testval’, dest=’testval’, help=’Specify a value’)
IronWatin.main(options=opts)
</code>
Thanks to IronPython, we can make use of our developers’ and QA engineers’ Python knowledge to get the up and running with writing integration tests using WatiN rapidly instead of trying to overcome the hump of teaching/training with a new language.
Deployment Notes: We’re using IronPython 2.6rc1 and building WatiN from trunk in order to take advantage of some recent advances in their Firefox/frame support. We’ve not tested IronWatin, or WatiN at all for that matter, anywhere other than Windows XP.
This weekend I finally got around to downloading IronPython 2.6rc1 to test it against the upcoming builds of Mono 2.6 preview 1 (the version numbers matched, it felt right). Additionally in the land of Mono, I’ve been toying around with the IKVM project as of late, as a means of bringing some legacy Java code that I’m familiar with onto the CLR. As I poked in one xterm (urxvt actually) with IKVM and with IronPython in another, a lightbulb went off. What if I could mix different languages in the same runtime; wouldn’t that just be cool as a cucumber? Turns out, it is.
After grabbing a recent release (0.40.0.1) of IKVM, I whipped up a simple Test.java file:
I compiled Test.java to Test.class then to Test.dll with ikvmc (note: this is using JDK 1.6); in short, Java was compiled to Java bytecode and then to CIL:
Once you have a DLL, it is fairly simple to import that into an IronPython script thanks to the clr module IronPython provides. It is important to note however, that IKVM generated DLLs will try to load other DLLs at runtime (IKVM.Runtime.dll for example) so these either need to be installed in the GAC or available in the directory your IronPython script is running in.
Here’s my sample test IronPython file, using the unittest module to verify that the compiled Java code is doing what I expect it to:
When I run the IronPython script, everything “just works”:
% mono IronPython-2.6/ipy.exe IkvmTest.py
.
----------------------------------------------------------------------
Ran 1 test in 0.040s
OK
%
While my Test.java is a fairly tame example of what is going on here, the underlying lesson is an important one. Thanks to the Mono project’s CLR and the advent of the DLR on top of that we are getting closer to where “language” and “runtime” are separated enough to not be interdependent (as it is with CPython), allowing me (or you) to compile or otherwise execute code written in multiple languages on a common (language) runtime.
Once upon a time I was lucky enough to take an “Intro to C++”
class taught by none other than
Bjarne Stroustrop
himself, while I learned a lot of things about what makes C++ good and sucky
at the same time, he also taught a very important lesson: great engineers are lazy.
It’s fairly easy to enumerate functionality in tens of hundreds of lines of poorly
organized, inefficient code, but (according to Bjarne) it’s the great engineers
that are capable of distilling that functionality into it’s most succinct
form. I’ve since taken this notion of being “ultimately lazy” into my
professional career, making it the root answer for a lot of my design decisions
and choices: “Why bother writing unit tests?” I’m too lazy to fire up the whole
application and click mouse buttons, and I can only do that so fast; “Why do you only
work with Vim in GNU/screen?” I can’t be bothered to set up a new slew of terminals
when I switch machines, and so on down the line.
Earlier this week I found another bit of manual work that I shouldn’t
be doing and should be lazy about: building. The local build is something
that’s common to every single software developer regardless of language, Slide being a Python shop,
we have a bit more subtle of a “build”, that is to say, developers implicitly
run a “build” when they hit a page in Apache or a test/script. I found myself
constantly switching between two terminal windows, one with my editor
(Vim) and one for running tests and other scripts.
Being an avid Hudson user, I decided I’d give
the File system SCM
a try. Very quickly I was able to set up Hudson to poll my working directory and
watch for files to change every minute, and then run a “build” with some tests
to go with it. Now I can simply sit in Vim all day and write code, only
context-switching to commit changes.
Setting up Hudson for local continuous integration is quite simple,
by visiting hudson-ci.org you can download
hudson.war which is a fully self contained
runnable version of Hudson, you can start it up locally with java -jar hudson.war.
Once it’s started, visit http://localhost:8080 and you’ve find
yourself smack-dab in the middle of a fresh installation of Hudson.
First things first, you’ll need the File System SCM plugin from the Hudson Update
Center (left side bar, “Manage Hudson” > “Manage Plugins” > “Available” tab)
After installing the plugin, you’ll need to restart Hudson, then you can create your
job, configuring the File System SCM to poll your working directory:
Of course, add the necessary build steps to build/test your software as well, and
you should be set for some good local continuous integration. Once the job is saved,
the job will
poll your working directory for files to be modified and then copy things over to
the job’s workspace for execution.
After the job is building, you can hook up the RSS feed
(http://localhost:8080/rssLatest) to
Growl or some
other form of desktop notifier so you don’t even have to move your eyes to know whether
your local build succeeded or not (I use the “hudsonnotify” script for Linux/libnotify
below).
By automating this part of my local workflow with Hudson I can take advantage of a few things:
I no longer need to context switch to run my tests
I can make use of Hudson’s nice UI for visually inspecting test results as they change over time
I have near-instant feedback on the validity of the changes I’m making
The only real downside I can think of is no longer having any excuse for checking
in code that “breaks the build”, but in the end that’s probably a good thing.
Instead of relying on commits, you can get near-instant feedback on your changes
before you even get things going far enough to check them in, tightening the feedback
loop on your changes even further, very-very continuous integration. Your mileage may
vary of course, but I recommend giving it a try.
monomiscellaneous
I recently found myself toying around with a number of web frameworks (like Seaside) to get a good read on who's doing what in the web world outside of Python and Django, when I stumbled across the ASP.NET MVC Add-in for MonoDevelop. Though the new Vim keybindings are sweet, I still can't effectively get work done in MonoDevelop yet. What MonoDevelop does do however is support generating Makefiles for any given project, which allowed me to create some Makefiles for an ASP.NET MVC project I had created in MonoDevelop, and port those Makefiles over to fit my NAnt and Vim-based workflow.
Along with building the necessary DLLs, I prefer to use my NAnt scripts to fire up the NUnit console and fire up a development instance of XSP to test my web applications out. All said and done this fairly basic script does the job; I typically run it with:
It seems every time @jasonrubenstein,
@ggoss3, @cablelounger
and I sit down to have lunch together, we invariably sway back and forth between
generic venting about “work stuff” and best practices for
doing aforementioned “work stuff” better. The topic of “reusable code” came up
over Mac ‘n Cheese and beers this afternoon, and I felt it warranted “wider
distribution” so to speak (yet-another-lame-Slide-inside-joke).
We, Slide, are approaching our fourth year in existence as a
startup which means all sorts of interesting things from an investor standpoint,
employees options are starting to become fully-vested and other mundane and boring
financial terms. Being an engineer, I don’t care too much about the stocks and such,
but rather about development; four years is a lot from a code-investment
standpoint (my bias towards code instead of financial planning will surely bite me
eventually). Projects can experience bitrot, bloating (read: Vista’ing) and a myriad
other illnesses endemic to software that’s starting to grow long in the tooth.
At Slide, we have a number of projects on slightly different trajectories and timelines,
meaning we have an intriguing cross-section of development histories representing
themselves. We are no doubt experiencing a similar phenomenon to Facebook, MySpace, Yelp and a number of other “startups” who match this same age group of 4-7
years. Just like our bretheren in the startup community, we have portions of code
that fit all the major possible categories:
That which was written extremely fast, without an afterthought to what would happen when it serve tens of millions of users
That which was written slowly, trying to cater to every possible variation, ultimately to go over-budget and over-schedule.
That which has been rewritten. And rewritten. And rewritten.
Then the exceptionally rare, that which has been written in such a fashion that it has been elegantly extended to support more than it was originally conceived to support.
In all four cases, “we” (whereas “we” refers to an engineering department) have
invested differently in our code portfolio depending on a number of factors and
information given at the time. For example, it’s been a year since Component X was
written. Component X is currently used by every single product The Company owns, but
over the past year it’s been refactored and partially rewritten each time a new
product starts to “use” Component X. In its current state, Component X’s code reads
more like an embarrasing submission to The Daily WTF with
its hodge-podge of code, passed from team to team, developer to developer, like some
expensive game of “Telephone” for
software engineers. After the fact, it’s difficult and not altogether helpful to
try to lay blame with the mighty sword of hindsight, but it is feasible to identify the
reasons for the N number of developer hours lost fiddling, extending, and refactoring
Component X.
Was the developer responsible for implementing Component X originally aware of the potentially far reaching scope of their work?
Was the developer given an adequate time frame to implement a proper solution, or “this should have shipped yesterday!”
Did somebody pass the project off to an intern or somebody who was on their way out the door?
Were other developers in similar realms of responsibility asked questions or for their opinions?
Is/was the culture proliferated by Engineering Leads and Managers encouraging of best practices that lead to extensible code?
I’ve found, watching Slide Engineering culture evolve, that the majority of libraries
or components that go through multiple time/resource-expensive iterations tend to have
experienced shortcomings in one of the five sections above. More often than not,
a developer was given the task to implement Some Thing. Simple enough, Some Thing
is developed with the specific use-case in mind, and the developer moves on with their
life. Three months later however, somebody else asks another developer, to add Some Thing
to another product.
“Product X has Some Thing, and it works great for them, let’s incorporate Some Thing into Product Y by the end of the week.”
Invariably this leads to heavy developer drinking. And then perhaps some copy-paste,
with a dash of re-jiggering, and quite possibly multiple forks of the same code. That
is, if Some Thing was not properly planned and designed in the first place.
Working as a developer on products that move at a fast pace, but will be around for
longer than three months is an exercise in investment strategy (i.e. managing
technical debt).
What makes great Engineering Managers great is their ability to determine when and where to invest
the time to do things right, and where to write some Perl-style write-only code (zing!).
What makes a startup environment a more difficult one to work on your “code portfolio”
is that you don’t usually know what may or may not be a success, and in a lot of cases
getting your product out there now is of paramount importance. Unfortunately
there isn’t any simple guideline or silver bullet, and there is no bailout, if
you invest your time poorly up front, there will be nobody to save you further down
the line when you’re staring an resource-devouring refactor in its ugly face.
Where do you invest the time in any given project? What will happen if you shave a
few days by deciding not to write any tests, or documentation. Will it cost you
a week further down the road if you take shortcuts now?
Since moving to the infrastructure team at Slide
I’ve found the rate at which my software gets deployed has plummeted, while the
quantity of the code that I am deploying to the live site has sky-rocketed. When
on an applications-team within Slide, code is typically pushed in small incrememnts a few
days a week, if not daily. This allows for really exciting compact milestones that
make more fine-grained analysis achievable, post-push for product management and metrics
purposes. On the infrastructure team however, the requirements are wholly different,
the “fail-fast, ship-now” mentality that prevails when doing user-facing web application
development just does not work in infrastructure. The most important aspects of building
out infrastructure components are stability and usability, our “customers” are the rest of
engineering, and that has a definite effect on your workflow.
Code Review
One of the things that @jasonrubenstein and I always
did when we worked together, was occasional code review. In the majority of cases, our “code review” sessions
were more or less rubber duck debugging, but
occasionally it would escalate into more complex discussions about the “right way” to do something.
When you’re writing infrastructure software for services that are handling tens of millions of users
the notion of “code review” goes from being optional to being absolutely required. Discussions
are had on the correctness or performance characteristics of database indexes, the necessity of some
objects instantiating default values of attributes or having them lazily load, or debating garbage
collection of objects while meticulously watching memory consumption.
For one of my most recent projects, I was working on something in C,
a rarity at Slide since we work with managed code in Python the majority of the
time. As the project neared completion, I counted roughly two or three hours of code review time
dedicated by our Chief Architect. The attention to detail paid to this code was extremely high,
as the service was going to be handling millions of requests from other levels of
the Slide infrastructure, before getting cycled or restarted.
A particularly frustrating aspect of code review by your peers is that a second set of eyes not only
will find problems with your code, but will likely mean refactoring or bug fixes, more work.
In my case, whenever a bug or stability issue was discovered, a test needed to be
written for it to make sure the bug did not present itself again, the workload would be larger than
if I had just fixed the bug and moved on with my life.
Testing, oh the testing
If you expect to write an API, have it stablize, and then be used, you must write test cases for
it. I’m not a TDD “nut”, I actually hate
writing test cases, I absolutely abhor it. Writing test cases is responsible and the adult
thing to do. In my experience, it can also be tedious and usually comes as a result of
finding flaws in my own software. The majority of tests that I find myself writing are admissions of
defeat, admitting that I don’t crap roses and by george, my code isn’t perfect either.
On the flipside however, I hate debugging even more. Stepping through a call stack is on par
with waterboarding in my book, torture. Which means I’m more than willing to tolerate writing tests
so long as it means I can be certain I will be cutting down on the time spent being tortured with
either pdb or gdb.
In almost every situation where I’ve written tests properly, like the responsible developer that
I am, I find them saving me at some point. It might be getting late, or I’m just feeling a little
cavalier, but tests failing almost always indicates that I’ve screwed something up I shouldn’t have.
Additionally, now that the majority of my projects are infrastructure-level projects, the tests I write serve
a second “undocumented” purpose, they provide ready-made examples for other developers on how to
use my code. Bonus!
The more and more code I write, the more amazed I am at the pushback against testing in general,
there exists decent libraries for every language imaginable (well, perhaps BrainfuckUnit doesn’t
exist), and its sole purpose (in my opinion) is to save develpoment time, particularly when coupled
with a good continuous integration server. Further to that effect, if you’re
building services for other developers to use, and you’re not writing tests for it, you’re not only
wasting your time and your employer’s money, but the time of your users as well (read: stop being a jerk).
Sure there are a lot of articles/books/etc about writing stable code, but in my opinion, solid code
review and testing will stablize your code far more than any design pattern ever will.
The first couple weeks of the Facebook platform were sheer insanity, determined to one-up our
competitors Rock You!, Slide acquired SuperPoke! and the three engineers that wrote it,
