Welcome to my blog where I write about software
development, cycling, and other random nonsense. This is not
the only place I write, you can find more words I typed on the Buoyant Data blog, Scribd tech blog, and GitHub.
While I don’t carry a decent camera around with me, I’ve found that the camera in my Samsung Jack is usually decent enough if I can hold my hands still (rare). Ended up snapping a few photos today and wanted to post them with a little bit of story behind them.
My bike route to the office takes me up and down Page St. which runs into Market St. just on the other side of Van Ness Ave. I never saw the accident or another car, I just saw a couple of cops and tow-truck guys standing around this flipped car on the turn from inbound Market St. onto Franklin St. I didn’t see any damage on the vehicle so I’m still quite puzzled how it was flipped and which direction it could have possibly been heading.
I picked these two figures up from Kid Robot for Apture’s Christmas Secret Santa, they ended up going to Josh who didn’t take them with him for whatever reason. I’ve since moved them to a more visible location (right below the Hudson dashboard).
Given the cloud cover that I normally ride home through I was quite surprised to see clear skies overTwin Peaks on my ride home. Of course, it is still February, so the clouds were still there, creeping along the ground.
Earlier this week I was checking out Pygame, pondering what I could possibly build with it that could keep me motivated enough to finish it. Motivation would like be the primary problem for me with any amount of game programming; I’m not a gamer, I don’t harbor a dislike of games, they’re just not something I typically spend time playing (I do like to play “haggard late night open-source hacker” though, that’s a fun one). Friday night I stumbled across an idea, ET likes to play (casual) games, perhaps we could write a game together; ask any engineer at EA or Ubisoft, there’s nothing more romantic than working on a game.
Talking over the idea with ET on the ride home from the office, we talked about creating a typing-oriented game and started to brainstorm. The tricky aspect of a typing-oriented game is you have to walk the fine line of “educational gaming”, that is to say, the game’s goal is not to teach the player how to type. That sucks. Contrasted to some other games where the means of progressing in some games is by solving a puzzle, killing noobs in others, in this game we wanted the player to progress through levels/situations with their typing ability (ET finds this fun, we do not have this in common).
Over pizza we discussed more about how the levels would work, I decided that I wanted to use stories/articles instead of random words for the “content” of the game. We settled on a couple fundamental concepts: the player would earn coins by correctly completing a words as the scrolled from right to left (similar to a ticker tape), they would lose coins if they made a mistake or could not keep up. After a player completed a level (i.e. a “story”) they would find themselves in a “store” of sorts, where they could purchase “tools” for future levels with their coins. The tools we decided would be a very important, as the player reached their upper bound of typing speed the utility of these various tools would necessary as a means of strategically conquering the level. One of the few things we didn’t particularly cover was the “end game”, whether the player would simply play increasingly more difficult levels (a la Tetris) or if they could actually “beat” the game. With at least the basics of the concept sketched out, it was time to start writing some code.
Starting with Pygame
It’s incredibly important to mention that I’ve never programmed a game before. Never-ever. From my work with network programming I was already familiar with the concept of the run-loop that’s pretty core to Pygame, but I had never really made use of any to animate objects on the screen or deal with handling any kind of events from mouse movements to key presses, etc. Fortunately I’m already a professional Python developer, so writing code wasn’t the difficult part so much as laying it out. Orienting things into classes to handle separate components such as animating text (which is a painful in Pygame, in my opinion) to keeping track of user-input.
Animating text across the screen wasn’t particularly difficult, with Pygame you first create your primary “surface” (i.e. the window) and then you can render things onto that surface. With text, you end up rendering a surface which contains your text, “hello world” which you then place onto the primary surface. Easy peasy thus far:
import pygame
surface = pygame.display.set_mode((640, 680), pygame.HWSURFACE | pygame.DOUBLEBUF)
font = pygame.font.SysFont('Courier', 42)
### render(text, antialias, rgb color tuple
font_surface = font.render('hello world', 0, (0, 0, 0))
### draw `font_surface` onto `surface` at (x=0, y=0)
start_x, start_y = 0, 0
surface.blit(font_surface, (start_x, start_y))
while True:
### Holy runloop batman
pygame.display.update()
That was fun, I now have “hello world” rendered onto my screen, now to animate I suppose I’ll just render font_surface a little further right every iteration of the runloop, i.e.
while True:
### Holy runloop batman
surface.blit(font_surface, (start_x, start_y))
start_x += 0.5
pygame.display.update()
This blurs the text however, so I then changed to:
while True:
### Holy runloop batman
surface.blit(font_surface, (start_x, start_y))
surface.fill((0, 0, 0))
start_x += 0.5
pygame.display.update()
This will cause the (primary) surface to be repainted (washed over) every iteration of the runloop ensuring that the text will properly animate, drawing the text in one spot, wiping the surface then drawing it slightly further to the left, resulting in the scrolling animation. All’s fine and good until you determine that you want to have other elements on the screen and you also don’t want to redraw them every time around the carousel. I then discovered how to “fill” just one particular rectangle on the surface, i.e. the rectangle behind the text:
text_w, text_h = font.size('hello world')
while True:
### Holy runloop batman
surface.blit(font_surface, (start_x, start_y))
surface.fill((0, 0, 0), rect=pygame.Rect(start_x, start_y, text_w, text_h))
start_x += 0.5
pygame.display.update()
Once I was able to get text properly scrolling across the screen, the rest of the afternoon of hacking was far easier. My confidence in my ability to grok Pygame in order to do what I wanted. I then set forth organizing my code into some logical classes, for example I created a LetterSpool class which would record the user’s progress through the current word, rendering it at the bottom-center of the screen and firing an event when the user hit the space bar (denoting the word “complete”), additionally I wrapped my text animation code into AnimatedWord so I could easily string words together to scroll across the screen in conjunction similar to a textual screensaver.
Not a whole lot more to write about with regards to my progress today, hooked up some music and basic sound effects which was trivial after looking at some sample code. Next I need to start addressing some more fundamentals for user-interaction: scoring and level-changing.
You can track the progress of the game “Typy” (pronounced: typey) on GitHub
My New Year’s resolution this year was incredibly generic insofar that I merely wanted to “write more.” No qualifications for what kind of writing that entailed, I simply want to become a better writer (or blogger), with technical subjects in particular I’d like to get better at writing in a fashion that is interesting, parse-able by novices and has sufficient “depth” to interest more technical readers. I’m not sure if I can define what being a “better writer” will entail or how I’ll know when I’m there, so for now I’m just trying to write good content. Considering my last post didn’t even pretend to ride the fence between opinionated-article and full-on rant, I think it’s safe to say that in order to accomplish my goal I need more venues for writing and more topics to write about.
One of those venues, which I’ve linked to before is the Apture Blog; I have written for the company blog already this year and chances are I will have another few posts go up as we tackle some of the technical challenges we’re currently facing (you can view my posts here). Unfortunately there’s only so many articles I can write for the Apture Blog without giving away any confidential information or turning it completely into a technical blog (hint: it’s not).
Looking around at a few of the open source communities that I’m involved in, two groups stick out: Eventlet and Hudson. Eventlet already has a blog and I’m certain my usage of Eventlet is not steady enough to warrant any kind of authoritative posts on the subject. The other, Hudson, is something I’ve used on a daily basis for almost a year and a half. Not only that, I run the @hudsonci twitter account and founded the #Hudson channel on Freenode, I’ve also tried my hand at developing some plugins for Hudson (which is written in Java). Suffice to say, I’m quite the little Hudson cheerleader.
When I floated the idea of an “official” blog for Hudson, which I would help drive, to Kohsuke and some other “core” developers of Hudson, the idea was well received and I set off getting Drupal configured, writing some preliminary content and getting ready for a launch of Continuous Blog. While my writing contributions thus far to Continuous Blog have been sparse, I’ve gotten to play the delightful role of Editor which is an entirely different experience unto itself.
I’m looking forward to seeing how this develops, I might end up writing for a few other blogs depending on interest and time, but for now my shenanigans can be found on:
There are few things I truly enjoy in life, things that warm my heart and make me smile from ear to ear, things like hot oatmeal and coffee, cell coverage in San Francisco, back scratches and not dying. If I didn’t commute every single day on my bicycle, I’d likely put “bike riding” on the list too but it’s hard to be ecstatic about something you start and finish every workday with (except for maybe some recreational yelling). For about 25 minutes every morning and 25 minutes every night, I’m a cyclist in San Francisco, nothing terribly unique, I am one of many cyclists in the city during rush hour and I have been noticing some things lately.
First of all, let me address my fellow cyclists. Some of you (you hopefully know who you are) are unadulterated, complete fucking-assholes. I understand that you don’t need a license to ride a bicycle but you still have to obey the rules of the road. Among other things this means you should be courteous to those you’re sharing the road with which includes:
Using hand signals
Yielding (you can use the dictionary on the iPhone that you’re pulling out at intersections to look up what that word means). Despite the ironic t-shirt you’re wearing underneath your hoodie, you are not “King of the World”.
Riding on streets safe for bicycle use. I know you love to ride down Oak St. and Van Ness in rush hour but it does nothing but piss drivers off. There are typically less trafficked streets adjacent to busy streets which are preferred for cyclist use (such as Page and Market St.).
Stopping at stop lights. I’m sure Gavin and the SFPD have given you an exemption from basic traffic laws but the drivers coming into the intersection don’t know that. You are making things dangerous for everybody around you on the road; stop it.
Given the hours I work, I’m typically riding home in the dark, where I’ve noticed a group of people who don’t share the same love of “not dying” as I do. I’m going to call them the “Suicide Cyclists.” This group of morons never cease to disappoint, without fail you’ll see one or two of them riding down a busy street (like Market St.) with no lights and no helmet. Some vindictive part of me really would like to see a car accidentally hit one of them. I don’t want them to seriously injure or kill anybody, just a light enough tap to send them to the ground and bump their head just hard enough to knock some god-damn sense into it.
There are parts of my bike ride where I might even crash into one of these Suicide Cyclists, places where I’m riding on a two-way bike path (through a poorly lit park) or where I’m turning at an intersection. If through some unfortunate turn of events we crash into each other, the cyclist wearing a dark sweatshirt with no helmet or lights on their bicycle, and me, with my plethora of lights and hand signals, you can be certain that the next thing that’s going to happen is assault with a U-Lock against some poor schmuck who’s too stupid to ride on the right side of the bike path in the dark.
To be honest, I can deal with the immense number of drivers not paying enough attention to the bike lanes, red lights or speed limits, the majority of the time cars do not cut through intersections, sidewalks, parks and everywhere else. Their trajectories are far more predictable than some jerk riding his fixie with his corduroy bike cap where ever he damn well pleases.
Some users of Hudson have already started to notice a subtle addition to the latest release, 1.343, a new background watermark image.
The commit message (r26728) from Kohsuke, the incredibly talented founder and maintainer of the Hudson project, adds a bit of sadness to the whole affair:<blockquote>In tribute to Sun Microsystems and all my colleagues who had to go today.
I hope the community would forgive me for doing this. </blockquote>
Given the incredible speed at which the tech industry grows and moves, it’s easy to forget that there are a number of talented engineers that have spent their careers at Sun building technologies that have helped change the face of modern computing, regardless of whether or not Sun could figure out how to sell them: SunOS/Solaris, Java, DTrace, SPARC 64-bit chips, Sun Grid Engine, JRuby, the W3C XML specification, ZFS, OpenOffice (acquisition), MySQL (acquisition), and VirtualBox (acquisition).
As a corporation, I personally think Sun was a failure, as a foundation of engineering in Silicon Valley, I think Sun has been quite successful.
To those that are being pushed out as part of the merger with Oracle, I want to sincerely thank you for your contributions to computing and wish you the best of luck.
Here’s the “full” version of the image, which I found via @jtnl’s TwitPic stream:
One of my most favorite sites on the internet, reddit, took some downtime this evening while doing some infrastructure (both hardware and software) upgrades. On their down-page, the reddit team invited everybody to join the #redditdowntime channel on the Freenode network, ostensibly to help users pass the time waiting for their pics and IAMAs to come back online.
Shortly after reddit started their scheduled outage, I joined the channel to pass the time while I debated what I should do with my evening. Within minutes the channel was flooded with a number of users, varying between spouting reddit memes in caps. link-spamming or engaging in casual chit-chat. I complained to one of the ops and fairly well-known-to-redditors employee: jedberg about the lack of moderation and he nearly instantly gave me +o (ops) in the channel. Not one to take my ops duty lightly, I started kicking spammers, warning habitual caps-lock users and tried to keep things generally civil through the deluge of messages consuming the channel.
Towards the end of the scheduled outage, some automated link-spamming started to appear and once it started it triggered more and more link-spamming. Clearly whatever was behind the bit.ly link was responsible for the self-propagating nature of the spamming. While the other moderators and myself tried to keep up with banning people I used wget to fetch the destination of the clearly malicious bit.ly URL to determine what we were dealing with. What I found is one of the more clever bits of JavaScript I think I’ve seen in recent months.
After bringing the site back up for a few minutes, reddit had to take it back down after noticing some problems with the upgrade, so another flood of users filled into the #redditdowntime channel and the link-spamming got worse. The most interesting aspect of the JavaScript in the code snippet below is how simple it is, I’ve commented it up a bit to help explain what’s actually going on:
Yesterday I was pointed to this post on ReadWriteWeb, suggesting that Facebook should acquire Apture next. Being an Apture employee, I would like to take some time to fuel the rumormill with these COMPLETELY TRUE (read: false) rumors:
I’m not going to cross-post but I wrote a little something on the Apture Blog about some of the things we’ve been doing lately to scale up with Django among other things. I suppose over the coming days I’ll have to write a few posts here getting into the nitty-gritty about Spawning vs. Apache and so on, but it’s a good start.
These days, the majority of my day job revolves around working with Apture’sDjango-based code which, depending on the situation, can be a blessing or a curse. In some of my recent work to help improve our ability to scale effectively, I started swapping out Apache for Spawning web servers which can more efficiently handle large numbers of concurrent requests. One of the mechanisms by which Spawning accomplishes this task, is by using eventlet’stpool (thread pool) module in addition to some other clever tricks. With Apache, we used pre-forked workers to accomplish the work needed to be done and while still using forked child processes with Spawning, threading was also thrown into the mix, that’s when “shit got real” (so to speak).
We started seeing sporadic, difficult to reproduce errors. Not a lot, a trickle of exception emails throughout the day. Digging deeper into some of the exceptions, careful stepping through Apture code, into Django code and back again, I started to realize I had thread-safety problems. Shock! Panic! Despair! Lunch! Disappointment! Shock! I felt all these things and more. I’ve long lamented the number of globals used in Django’s code base but this is the icing on the cake.
Apparently Django’s threading problems are sufficiently documented in a few places. Using a slightly older version of the Django framework certainly doesn’t help but it doesn’t appear that recent releases (1.1.1) can guarantee thread-safety anyways. I think it’s safe to assume the majority of Django framework users are not using threaded web servers in any capacity, else this would have become a far larger issue (and hopefully of been fixed) by now. From NoReverseMatch exceptions, to curious middleware problems to thread-safety issues in the WSGI support layer, Django has potholes lying all along the road to multithreadedness.
Lately I’ve fallen in love with a couple of fairly simple but powerful
technologies: haproxy and WSGI (web server gateway interface). While the latter
is more of a specification (PEP 333) the concepts it puts forth have made my life
significantly easier. In combination, the two of them make for a powerful combination
for serving web applications of all kinds and colors.
HAProxy is a robust, reliable piece of load balancing software that’s very easy
to get started with, For the uninitiated, load balancing is a common means of distributing
the load of a number of inbound requests across a pool of processes, machines, clusters and so on.
Whenever you hit any web site of non-trivial size, your HTTP requests are invariably transparently
proxied through a load balancer to a pool of web machines.
I started looking into haproxy when I began to move Urlenco.de
away from my franken-setup of Lighttpd/FastCGI/Mono/ASP.NET to a pure Python stack.
After poking around some articles about haproxy I discovered it can be used for virtual hosts
as well as simple load balancing. Using a haproxy’s ACLs feature (see Section 7 in the
configuration.txt), you can
redirect requests to one backend or another. While my “virtual hosting” with haproxy is using
the ability to inspect the HTTP headers of inbound requests, you can use a number of different
criterion to determine the right backend for serving a request: url matching, request method matching
(GET/POST), protocol matching (haproxy can load balance any kind of TCP connection) and so on.
WSGI (pronounced: whiskey) comes into play on the backend side of haproxy, using the
eventlet.wsgi module which provides a WSGI interface I can build web applications very
quickly, test them and deploy them. When deployed, I can run them as “nobody” in userspace on
the server, binding to some higher numbered port (i.e. 8080) and haproxy will do the work routing
to the appropriate WSGI process.
Below is a simple haproxy configuration that I’m using to run Urlenco.de and
a site for my wedding and many more as soon as I finish them. The section to note is frontend http-in in which the ACLs are defined for the different virtually hosted domains and the conditionals for selecting a backend based on those ACLs.
global
maxconn 20000
ulimit-n 16384
log 127.0.0.1 local0
uid 200
gid 200
chroot /var/empty
nbproc 4
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http-in
bind *:80
acl is_urlencode hdr_end(host) -i urlenco.de
acl is_wedding hdr_end(host) -i erinandtylerswedding.com
use_backend urlencode if is_urlencode
use_backend wedding if is_wedding
default_backend urlencode
backend urlencode
balance roundrobin
cookie SERVERID insert nocache indirect
option httpchk HEAD /check.txt HTTP/1.0
option httpclose
option forwardfor
server Local 127.0.0.1:8181 cookie Local
backend wedding
balance roundrobin
cookie SERVERID insert nocache indirect
option httpchk HEAD /check.txt HTTP/1.0
option httpclose
option forwardfor
server Local 127.0.0.1:8081 cookie Local
There’s quite a big advertising blitz in San Francisco for the “Book of Eli” movie, which as far as I can tell is another in a line of quasi-religious films (fortunately Tom Hanks doesn’t star in this one). I now see a billboard that’s a derivation of this on my ride home from work:
To be honest, I was already not going to see this movie but their marketing campaign has hammered the final nail in the cross (er, coffin). Perhaps I’m far more woeful of religion after seeing too many documentaries like “Jesus Camp” and watching clips of Pat Robertson.
In a world of spiteful neo-conservatives hijacking public discourse on important issues with nonsense about the gays, abortion and anything else that can be misconstrued as “christian values.” A world where radical sects of Islam kidnap, murder and terrorize; a world where no religion is without blood on their hands, the billboard is technically right.
Since joining Apture, I’ve primarily concerned myself with lower-level backend code and services, including the machines that our site runs on. While not a drastic departure from my role on the server team at Slide, there are a few notable changes, the largest of which being root. Given the size of Slide’s operations team, a team separate from the “server team” (the latter being developers), my role did not necessitate server management only occasional monitoring. Apture is a different can of beans, we’re simply too small for an operations team, so we work with Contegix to maintain a constant watchful eye on our production environment. Self-assigning myself the “backend guy” hat means server maintenance and operations are part of my concern (but not my focus) since the “goings on” of the physical machines will have a direct impact on the performance and level of service my work can ultimately provide to end users.
Last week while planning some changes we can make to our Django-based production environment that will help us grow more effectively, Steven pointed out that we were going to see an influx of usage today (Jan. 4th) given the large number of users returning to the internet after their holiday vacation. Over the weekend I dreaded what Monday would bring, unable to enact any changes to stave off the inevitable in time.
This morning, waking up uncharacteristically early at 7 a.m. PST, bells were already ringing. A 9 a.m. EST spike angered one of our database machines, by the time I got in the office around 8:10 a.m. PST more bells were ringing as the second wave of users once again angered the MySQL Dolphin Gods. With my morning interview candidate already on site, I furiously typed off a few emails to Contegix sounding the alarm, pleading for help, load balancer tweaks, configuration reviews, anything to help squeeze extra juice from the abnormally overloaded machines to keep our desired level of service up. Working with a few of the talented Contegix admins we quickly fixed some issues with the load balancer under utilizing certain machines in favor of others, isolated a few sources of leaked CPU cycles and discovered a few key places to add more caching with memcached(8).
As our normal peak (~9 a.m. PST to around lunchtime) passed, I started to breathe easier when alarms went of again. Once again, Contegix admins were right there to help us through one of our longest peak I’ve seen since joining Apture, 5:30 a.m. until around 4 p.m.
Survival was my primary objective waking up today but thanks to some initiative and good footwork by the folks at Contegix we not only survived but identified and corrected a number of issues detrimental to performance and discovered on of the key catalysts of cascading load: I/O strapped database servers (as MySQL servers starve for disk I/O, waiting requests in Apache drive the load on a machine through the roof).
I am admittedly quite smitten with Contegix’s work today, I became quite accustomed to KB and his ops team at Slide fixing whatever issues would arise in our production environment and it’s comforting to know that we have that level of sysadmin talent at the ready.
A picture is worth a thousand words; here’s a cumulative load graph from our production Ganglia instance:
What’s the coolest Python application, framework or library you have discovered in 2009?
While I didn’t discover it until the latter half of 2009, I’d have to say eventlet is the coolest Python library I discovered in 2009. After leaving Slide, where I learned the joys of coroutines (a concept previously foreign to me) I briefly contemplated using greenlet to write a coroutines library similar to what is used at Slide. Fortunately I stumbled across eventlet in time, which shares common ancestry with Slide’s proprietary library.
What new programming technique did you learn in 2009?
I’m not sure I really learned any new techniques over the past year, I started writing a lot more tests this past year but my habits don’t quite qualify as Test Driven Development just yet. As far as Python goes, I’ve been introduced to the Python C API over the past year (written two entire modules in C PyECC and py-yajl) and while I wouldn’t exactly call implementing Python modules in C a “technique” it’s certainly a departure from regular Python (Py_XDECREF I’m looking at you)
What’s the name of the open source project you contributed the most in 2009? What did you do?
Regular readers of my blog can likely guess which open source project I contributed to most in 2009, Cheetah, of which I’ve become the maintainer. I also authored a number of new Python projects in 2009: PyECC a module implementing Elliptical Curve Cryptography (built on top of seccure), py-yajl a module utilizing yajl for fast JSON encoding/decoding, IronWatin an IronPython-based module for writing WatiN tests in Python (supporting screengrabs as well), PILServ an eventlet-based server to do server-side image transformations with PIL, TweepyDeck a PyGTK+ based Twitter client and MicroMVC a teeny-tiny MVC styled framework for Python and WSGI built on eventlet and Cheetah.
What was the Python blog or website you read the most in 2009?
The Python reddit was probably the most read Python-related “blog” I read in 2009, it generally supercedes the Python Planet with regards to content but also includes discussions as well as package release posts.
What are the top three things you want to learn in 2010?
Python 3. After spending a couple weekends trying to get Cheetah into good working order on Python 3, I must say, maintaining a Python-based module on both Python 2.xx and 3.xx really feels like a nightmare. py-yajl on the otherhand, being entirely C, was trivial to get compiling and executing perfectly for 2.xx and 3.xx
NoSQL. Earlier this very evening I dumped a boatload of data out of PostgreSQL into Redis and the resulting Python code for data access using redis-py is shockingly simple. I’m looking forward to finding more places where a relational database is overkill for certain types of stored data, and using Redis instead.
Optimizing Python. With py-yajl Lloyd and I had some fun optimizing the C code behind the module, but I’d love to learn some handy tricks to making pure-Python execute as fast as possible.
A few months ago Kohsuke, author of the Hudson continuous integration server,
introduced me to the concept of the “pre-tested commit”, a feature of the TeamCity
build management and continuous integration system. The concept is simple, the build
system stands as a roadblock between your commit entering trunk and only after the
build system determines that your commit doesn’t break things does it allow the commit
to be introduced into version control, where other developers will sync and integrate
that change into their local working copies. The reasoning and workflow put forth by
TeamCity for “pre-tested commits” is very dependent on a centralized version control
system, it is solving an issue Git or Mercurial users don’t really run into. Those using
Git can commit their hearts out all day long and it won’t affect their colleagues until they
merge their commits with others.
In some cases, allowing buggy or broken code to be merged in from another developer’s Git
repository can be worse than in a central version control system, since the recipient of the
broken code might perform a knee-jerk git-revert(1) command on the merge! When you revert
a merge commit in Git, what happens is you not only revert the merge, you revert the commits
associated with that merge commit; in essence, you’re reverting everything you just merged in
when you likely just wanted to get the broken code out of your local tree so you could continue
working without interruption. To solve for this problem-case, I utilize a “pre-tested commit” or
“pre-tested merge” workflow with Hudson.
My workflow with Hudson for pre-tested commits involves three separate Git repositories: my local
repo (local), the canonical/central repo (origin) and my “world-readable” (inside the firewall) repo (public).
For pre-tested commits, I utilize a constantly changing branch called “pu” (potential updates) on the
world-readable repo. Inside of Hudson I created a job that polls the world-readable repo (public)
for changes in the “pu” branch and will kick off builds when updates are pushed. Since the content of
public/pu is constantly changing, the git-push(1) commands to it must be “forced-updates” since I am
effectively rewriting history every time I push to public/pu.
To help forcefully pushing updates from my current local branch to public/pu I use the following git alias:
% git config alias.pup "\!f() { branch=\$(git symbolic-ref HEAD | sed 's/refs\\/heads\\///g');\
git push -f \$1 +\${branch}:pu;}; f"
While a little obfuscated, thie pup alias forcefully pushes the contents of the current branch to the specified
remote repository’s pu branch. I find this is easier than constantly typing out: git push -f public +topic:pu
In list form, my workflow for taking a change from inception to origin is:
hack, hack, hack
commit to local/topic
git pup public
Hudson polls public/pu
Hudson runs potential-updates job
Tests fail?
Yes: Rework commit, try again
No: Continue
Rebase onto local/master
Push to origin/master
Using this pre-tested commit workflow I can offload the majority of my testing requirements to the build system’s cluster of machines instead of running them locally, meaning I can spend the majority of my time writing code instead of waiting for tests to complete on my own machine in between coding iterations.
This year my family celebrated the holidays in north Florida at my older sister’s house, fortunately for the location is just as difficult to get to by plane as my parent’s house, so I didn’t have to miss out on any air travel frustration. My trip to north Florida was very boring, my flight out of San Francisco left at 6 a.m. and I arrived in Jacksonville around dinner time (having slept the majority of the flight). The return trip was far more eventful, I left my sister’s house around noon to drive to Jacksonville (roughly an hour and a half trip), waited at the airport for my flight at 4 p.m., arrived in Miami at 5:30, waited for hours on a delayed flight, left Miami around 10 p.m., landed in San Francisco a hours later than anticipated, paid my exorbitant parking fee and sped home.
When I woke up the next day, I looked over at how bitchy and whiney my posts to Twitter from the previous day were. I don’t think I’m normally that big of a jerk but traveling alone, I needed to vent, often. (note: times listed are PST, for the majority of the trip I was in EST)
11:10 AM Still kind of amazes me how many young women in the south are running around with babies in tow.
11:34 AM How the news should cover this incident: Guy tries something on plane, passengers take him out. Post-9/11, TSA is pointless
12:57 PM Shit. My first flight today has propellers. Fucking propellers.
Suffice to say, I don’t think I’m flying American Airlines again for some time (or at all for that matter). The whole experienc to Florida and back was grueling to say the least; with the parking fees, baggage fees, meal fees, delays and endless hours breathing recycled air riddled with H1N1 and sneezes, I think I’m going to keep my feet on the ground for a while.
In a past life I traveled quite frequently, being categorically
poor as I often was, I tended to rely on the kindness of friends,
family and occasionally total strangers. After breaking the standing
record for longest-time-spent-on-Dave’s couch, I came to consider myself
a pretty decent house guest. More recently I spent this past week at my
older sister’s house with a swarm of other family members, as I cooked
breakfast for the family Saturday morning, I decided that I’m not only
a pretty decent house guest, I’m a pretty stinkin’ awesome guest who you
should invite over if you:
Feel like cooking a big dinner but don’t want to do the dishes
Are just sick and tired of cooking and really would like somebody else to make you something delicious
Feel the need to have a wide-ranging discussion regarding national and international politics with a mildly intelligent person
While I know that you’re not supposed to stay too long as a house guest,
I think the social rule comes from a long line of either unwelcome guests
or guests that just aren’t doing it right. Here are the rules I try to
follow whenever I find myself crashing on some kind person’s couch, floor
or air mattress.
Keep your things tidy
Very important, yes you’re likely traveling, the folks you’re staying with
understand that you don’t have a closet or dresser you can throw your
clothes in, it is very important however that you keep as much of your
belongings tidily stashed away in your suitcase. The extra effort goes
a long way in making your presence far less impactful on those hosting you.
Nobody likes a dirty home.
Offer to cook
Unless your hosts have more money than they should, chances are that they
have jobs and when they come home from those jobs they have to cook themselves
and their family dinner. Offering to cook goes a long way with a lot of people,
especially if you can actually cook. (note: cooking delicious food is not
difficult, but really just a test of your ability to read a recipe). Not
everybody will take you up on your offer, some people (myself included) find
cooking a good way to unwind after a day’s work. If you find yourself in this
situation, linger around the kitchen, socialize and try to be as helpful as
possible; an extra set of hands and eyes to watch a pot, or peel potatoes is
almost always appreciated.
Hang out, don’t cling
Most people enjoy having company, humans are inherently social animals and
having a house guest can be an nice change of pace for a lot of people. If
you’re traveling through, you’ll have to walk a fine line of hanging out with
the hosting party long enough to have fun together but not long enough to
make them feel smothered. The system I’ve always followed is to be occupied during the day and social with my hosts in the evening. This gives them a chance to have a normal workday or weekend, and gives me the chance to explore my current location on my own and have an adventure. This set up works quite well when traveling abroad since you get the opportunity to regale your hosts with tales of your adventure in their region over dinner (note; do not trash their city, people tend to have some amount of pride for their city/region/state).
Do chores
If you’re staying with any host for any time longer than a few days, it is highly
likely that some cleaning, vacuuming, laundry or dishes will need to be done. A good rule of thumb with chores is not to offer to help but just to help out where you can, a quick “let me give you a hand with that..” will do.
Nobody will turn down a helping hand when it comes to cleaning.
There are occasions when I’ve preferred hotels to crashing with friends or family, when I really need a good night’s rest and some quiet, but if you’re up for a good sociable experience, you really cannot beat crashing on a couch.
Just don’t borrow any money, they really hate that.
miscellaneous
My relationship with this cat is one of constant fluctuation, when he's quiet and/or asleep, he's the best cat in the world. When I come home and he vies for attention, he drives me nuts.
After running a Linux laptop for a number of years and
having mostly negative travel experiences from messing something up
along the way, this holiday season I think I’ve finally figured
out how to optimally travel with a Linux notebook. The following tips
are some of the lessons I’ve had to learn the hard way through trial and
error over the course of countless flights spanning a few years.
Purchase a small laptop or netbook
Far and away the best thing I’ve done for my travel experience thus far
has been the purchase of my new Thinkpad X200 (12.1”). My previous laptops include
a MacBook Pro (15”), a Thinkpad T43 (14”) and a Thinkpad T64 (14”). Invariably I have
the same problems with all larger laptops, their size is unwieldy in economy class
and their power consumption usually allows me very little time to get anything done
while up in the air. Being 6’4” and consistently cheap, I’m always in coach, quite
often on redeye flights where the passenger in front of me invariably leans their
seat back drastically reducing my ability to open a larger laptop and see the screen.
With a 12” laptop or a netbook (I’ve traveled with an Eee PC in the past as well) I’m able
to open the screen enough to see it clearly and actually type comfortbaly on it. Additionally,
the smaller screen and size of the laptop means less power consumption, allowing me to
use it for extended periods of time.
Use a basic window manager
Personally, I prefer XMonad, but I believe any simplistic window manager will save
a noticable number of cycles compared to the Gnome and KDE “desktop environments.”
Unlike Gnome, for example, XMonad does not run a number of background daemons to help
provide a “nice” experience in the way of applets, widgets, panels and menus.
Disable unneeded services and hardware
Reducing power consumption is a pretty important goal for me while traveling with
a Linux laptop, I love it when I have sufficient juice to keep myself entertained
for an entire cross-country flight. Two of the first things I disable before boarding
a plane are Wireless and Bluetooth via the NetworkManager applet that I run. If I’m on a
redeye, I’ll also set my display as dark as possible which not only saves power but also
eye strain. It’s also important to make sure your laptop is running its CPU in “power-save”
mode, which means the clockspeed of the chip is reduced, allowing you to save even more power.
Finally I typically take a look at htop(1) to see if there are any unneeded processes taking
up cycles/memory that I either don’t need or don’t intend to use for the flight. The flight
I’m currently on (Miami to San Francisco) I discovered that Chrome was churning some
unnecessary cycles and killed it (no web browsing on American Airlines).
Use an external device for music/video
If you’re like me, you travel with a good pair of headphones and a desire to not listen
to babies crying on the plane. I find a dedicated device purely for music can help avoid
wasting power on music since most devices can play for 12-40 hours depending on the device.
It’s generally better (in my opinion) to use your $100 iPod for music and your $2000 computer
for computing, that might just be personal bias though.
Load applications you’ll need ahead of time
I generally have an idea of what I want to do before I board a plane, I have a project
that I’d like to spend some time hacking on or something I want to write out or
experiment with. Having a “game plan” before I get onto the plane means I can load up
any and all applications while plugged in at the airport. This might be a minor power
saver but after I’ve lowered the CPU clockspeed and disabled some services, I certainly
don’t want to wait around for applications to load up while I sit idly in coach.
Update: As Etni3s from reddit points out, powertop(1) is a pretty handy utility for watching power consumption.
As I write this article, I’m probably an hour into my five and half hour flight and the
battery monitor for my X200 is telling me I have an estimated eight hours of juice
left.
Some time ago after reading a post on Eric Florenzano’s blog about hacking together support for Cheetah with Django, I decided to add “proper” support for Cheetah/Django to Cheetah v2.2.1 (released June 1st, 2009). At the time I didn’t use Django for anything, so I didn’t really think about it too much more.
Now that I work at Apture, which uses Django as part of its stack, Cheetah and Django playing nicely together is more attractive to me and as such I wanted to jot down a quick example project for others to use for getting started with Cheetah and Django. You can find the django_cheetah_example project on GitHub, but the gist of how this works is as follows.
For all intents and purposes, using Cheetah in place of Django’s
templating system is a trivial change in how you write your views.
After following the Django getting started
documentation, you’ll want to create a directory for your Cheetah templates, such
as Cheetar/templates. Be sure to touch __init__.py in your template
directory to ensure that templates can be imported if they need to.
Add your new template directory to the TEMPLATE_DIRS attribute
in your project’s settings.py.
Once that is all set up, utilizing Cheetah templates in Django is just
a matter of a few lines in your view code:
import Cheetah.Django
Note: Any keyword-arguments you pass into the Cheetah.Django.render()
function will be exposed in the template’s “searchList”, meaning you can
then access them with $-placeholders. (i.e. $greet)
With the current release of Cheetah (v2.4.1), there isn’t support for using pre-compiled Cheetah templates with Django (it’d be trivial to put together though) which means Cheetah.Django.render() uses Cheetah’s dynamic compilation mode which can add a bit of overhead since templates are compiled at runtime (your mileage may vary).
Those that have worked with my directly know I’m a tad
obsessive when it comes to imports in Python. Once upon a
time I had to write some pretty disgusting import hooks
to solve a problem and got to learn first-hand how gnarly
Python’s import subsystem can be. I have a couple coding
conventions that I follow when I’m writing Python for my
own personal projects that typically follows:
“strict” system imports first (i.e. import time)
“from” system imports second (i.e. from eventlet import api)
“local” imports (import mymodule)
local “from” imports (from mypackage import module)
In all of these sections, I like to list things alphabetically
as well, just to make sure that at no point are modules ever
doubley-imported. This results in code that looks clean (in
my humblest of opinions):
#!/usr/bin/env python
import os
import sys
from eventlet import api
import app.util
from app.models import account
## Etc.</code>
A module importing habit that absolutely drives me up the wall,
I was introduced to and told “don’t-do-that” by Dave: importing
symbols from modules; in effect: from MySQLdb import IntegrityError.
I have two major reasons for hating the importing of symbols, the
first one is that it messes with your module’s namespace. If the
symbol import above were in a file called “foo.py”, the foo module
would then have the member foo.IntegrityError. Additionally, it
makes the code more difficult to understand when you flatten the module’s
namespace out; 500 lines down in the file if you see acct_m = AccountManager()
as a developer new to the file you’ll have to go up to the top and figure
out where the hell AccountManager is actually coming from to understand
how it works.
As code with these sort of symbol-level imports ages, it becomes more and more
frustrating to deal with, if I need OperationalError in my module now I have
three options:
Update the line to say: from MySQLdb import IntegrityError, OperationalError
Add import MySQLdb and just refer to IntegrityError and MySQLdb.OperationalError
Add import MySQLdb and update all references to IntegrityError
I’ve seen code in open source projects that have abused the symbol imports
so badly that an import statement look like: from mod import CONST1, CONST2, CONST3, SomeError, AnotherClass
(ad infinium).
I think poor import style is a good indicator of how one can expect the
rest of the Python code to look, I cannot recall a single instance where I’ve
looked at a Python module with gross import statements and clean classes and functions.
from MySQLdb import IntegrityError, OperationalError, MySQLError, ProgrammingError, \
NotSupportedError, InternalError
