<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://brokenco.de//feed/by_tag/python.xml" rel="self" type="application/atom+xml" /><link href="https://brokenco.de//" rel="alternate" type="text/html" /><updated>2026-08-15T16:52:02+00:00</updated><id>https://brokenco.de//feed/by_tag/python.xml</id><title type="html">rtyler</title><subtitle>a moderately technical blog</subtitle><author><name>R. Tyler Croy</name></author><entry><title type="html">The async stall</title><link href="https://brokenco.de//2026/07/12/compute-on-coroutines.html" rel="alternate" type="text/html" title="The async stall" /><published>2026-07-12T00:00:00+00:00</published><updated>2026-07-12T00:00:00+00:00</updated><id>https://brokenco.de//2026/07/12/compute-on-coroutines</id><content type="html" xml:base="https://brokenco.de//2026/07/12/compute-on-coroutines.html"><![CDATA[<p>I am loving the proliferation of async/await-style cooperative multitasking
across different language ecosystems. My first job in San Francisco was
building with greenthreads in Python (2.4!) in an era where Stackless,
Greenlets, and Twisted were all fringe communities inhabited by oddballs like
myself. Nowadays it’s a little embarrassing to adopt a toolchain that doesn’t
support some form of I/O-evented cooperative multitasking. Node.js, Rust, and
Python all have native support, Ruby’s Fiber’s seem to have missed the mark but
receive points for effort. All of these tools allow developers to write
simple code that appears procedural but <em>isn’t actually</em> at runtime.</p>

<p>The old do-it-mostly-yourself approaches forced the idea of an “event loop” in to our code in a way that the newer async/await semantics hide. 
Deep under the
covers there is a
<a href="https://man.freebsd.org/cgi/man.cgi?query=kevent&amp;apropos=0&amp;sektion=0&amp;manpath=FreeBSD+15.1-RELEASE+and+Ports.quarterly&amp;format=html">kqueue</a>
or an
<a href="https://man.freebsd.org/cgi/man.cgi?query=epoll&amp;apropos=0&amp;sektion=0&amp;manpath=openSUSE+42.3&amp;format=html">epoll</a>
which is doing a <em>lot</em> of work for you, and the abstractions are hiding a for
loop that is iterating over file descriptors selecting whichever one is ready
to continue execution.</p>

<p>The naming of the package <code class="language-plaintext highlighter-rouge">asyncio</code> in Python helps convey that it’s for I/O —
network calls, disk reads, waiting on things. That framing is mostly right, but
it invites a failure mode I keep seeing: someone discovers <code class="language-plaintext highlighter-rouge">asyncio</code> solves
their concurrency problems and but as time goes on they loose track of how it
works, and start CPU-heavy work inside coroutines. It does not end well. The
event loop stalls, tail latencies balloon, and suddenly your health check is
timing out in Kubernetes while your process is contentedly crunching data.</p>

<p>Once <code class="language-plaintext highlighter-rouge">asyncio</code> bites, the knee-jerk reaction I have seen is to rip out the
<code class="language-plaintext highlighter-rouge">asyncio</code> code and go back to the comfortable domain of <code class="language-plaintext highlighter-rouge">multiprocessing</code> or
equivalent. Hiding from the problem rather than addressing it.</p>

<p><strong>We can have our cake</strong> while still using <code class="language-plaintext highlighter-rouge">asyncio</code> for CPU-heavy applications.</p>

<p>Andrew Lamb <a href="https://www.youtube.com/watch?v=FeqRdDG1Y7g">spoke about exactly this
problem</a> in the context of Tokio:
the fix isn’t to avoid the async runtime for CPU work, it’s to <strong>stop mixing your
latency-sensitive and compute-heavy work on the same executor</strong>.</p>

<p>The pattern that actually works
is a dedicated <code class="language-plaintext highlighter-rouge">ProcessPoolExecutor</code> for the heavy lifting, submitted via
<code class="language-plaintext highlighter-rouge">loop.run_in_executor()</code>, while your event loop stays unblocked and responsive.
Python still has a GIL so there may be some use-cases that still require multi-processing to achieve high levels of performance but a better segmentation of low-latency and compute heavy workload in the <code class="language-plaintext highlighter-rouge">asyncio</code> eventloop can go a long way.</p>

<p>Some tips to consider:</p>

<ol>
  <li>
    <p><strong>Don’t block the event loop — ever</strong> Async code should never spend a long
time without yielding. In Python, CPU-bound work in a coroutine starves the
event loop. Use <code class="language-plaintext highlighter-rouge">await asyncio.sleep(0)</code> periodically, or offload entirely.</p>
  </li>
  <li>
    <p><strong>Use a separate executor for CPU work</strong> I have seen errors where an
applications’ health check got blocked by CPU heavy work, leading to
seemingly interruptions of service as the container orchestrator killed
unresponsive containers.</p>

    <p>Run CPU tasks on a <em>separate</em> thread/process pool,
not the same one handling I/O. In Python: <code class="language-plaintext highlighter-rouge">loop.run_in_executor(executor,
fn)</code> with a dedicated <code class="language-plaintext highlighter-rouge">ProcessPoolExecutor</code>. Don’t share it with your I/O
event loop.</p>
  </li>
  <li>
    <p><strong>GIL means processes, not threads</strong> Unlike Tokio’s work-stealing thread
pool, Python threads don’t get true parallelism for CPU work due to the GIL.
Use <code class="language-plaintext highlighter-rouge">concurrent.futures.ProcessPoolExecutor</code> instead of <code class="language-plaintext highlighter-rouge">ThreadPoolExecutor</code>
for CPU-bound tasks.</p>
  </li>
  <li>
    <p><strong>Amortize overhead with chunked work</strong> For CPU-heavy workloads in Python
consider making larger batches when possible,. A method call with a single
row in a runtime like Python is going to have higher wasted overhead when
invoking that method 100k times in a tight loop compared to invoking a
method which is able to handle a batch of 100k rows <em>outside</em> of Python
(e.g. in a Rust or C extension).</p>
  </li>
  <li>
    <p><strong>Cancellation and shutdown are hard</strong> Kind of niche advice, but 
hard with custom schedulers are easy to get 99.9% right but corner cases
(shutdown, cancellation, draining) waste a bunch of time. Just use the
off-the-shelf schedulers. In Python, lean on <code class="language-plaintext highlighter-rouge">asyncio.Task.cancel()</code> and
<code class="language-plaintext highlighter-rouge">executor.shutdown(wait=True)</code> rather than rolling your own.</p>
  </li>
</ol>

<p>I love to advocate the use of
<a href="https://rust-lang.org">Rust</a> for lots of projects, but modern Python with
judicious use of <code class="language-plaintext highlighter-rouge">asyncio</code> and a lot of the more modern APIs available since
Python 3.10-ish make it a much better option for high-performance applications
with a low barrier to entry.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="software" /><category term="rust" /><category term="python" /><summary type="html"><![CDATA[I am loving the proliferation of async/await-style cooperative multitasking across different language ecosystems. My first job in San Francisco was building with greenthreads in Python (2.4!) in an era where Stackless, Greenlets, and Twisted were all fringe communities inhabited by oddballs like myself. Nowadays it’s a little embarrassing to adopt a toolchain that doesn’t support some form of I/O-evented cooperative multitasking. Node.js, Rust, and Python all have native support, Ruby’s Fiber’s seem to have missed the mark but receive points for effort. All of these tools allow developers to write simple code that appears procedural but isn’t actually at runtime.]]></summary></entry><entry><title type="html">Unclog the tubes; blocking detection in Eventlet</title><link href="https://brokenco.de//2010/08/28/unclog-the-tubes-blocking-detection-in-eventlet.html" rel="alternate" type="text/html" title="Unclog the tubes; blocking detection in Eventlet" /><published>2010-08-28T00:00:00+00:00</published><updated>2010-08-28T00:00:00+00:00</updated><id>https://brokenco.de//2010/08/28/unclog-the-tubes-blocking-detection-in-eventlet</id><content type="html" xml:base="https://brokenco.de//2010/08/28/unclog-the-tubes-blocking-detection-in-eventlet.html"><![CDATA[<p>Colleagues of mine are all very familiar with my admiration of <a href="http://eventlet.net">Eventlet</a>, a
Python concurrency library, built on top of <a href="http://pypi.python.org/pypi/greenlet">greenlet</a>, that
provides lightweight “greenthreads” that naturally yield around I/O points. For me, the biggest draw of Eventlet
besides its maturity, is how well it integrates with standard Python code. Any code that uses the built-in
<code class="language-plaintext highlighter-rouge">socket</code> module can be “monkey-patched” (i.e. modified at runtime) to use the “green” version of the socket
module which allows Eventlet to turn regular ol’ Python into code with asynchronous I/O.</p>

<p>The problem with using libraries like Eventlet, is that some Python code just <strong>blocks</strong>, meaning that
code will hit an I/O point and <em>not</em> yield but instead block the entire process until that network operation
completes.</p>

<p>In practical terms, imagine you have a web crawler that uses 10 “green threads”, each crawling a
different site. The first greenthread (GT1) will send an HTTP request to the first site, then it will yield
to GT2 and so on. If each HTTP request blocks for 100ms, that means when crawling the 10 sites, you’re going
to block the whole process, preventing anything from running, for a whole second. Doesn’t sound too terrible,
but imagine you’ve got 1000 greenthreads, instead of everything smoothly yielding from one thread to another
the process will lock up very often resulting in painful slowdowns.</p>

<p>Starting with Eventlet 0.9.10 “blocking detection” code has been incorporated into Eventlet to make
it far easier for developers to find these portions of code that can block the entire process.
<code type="python">
    import eventlet.debug
    eventlet.debug.hub_blocking_detection(True)
</code></p>

<p>While using the blocking detection is fairly simple, its implementation is a bit “magical” in that
it’s not entirely obvious how it works. The detector is built around signals, inside of Eventlet a signal
handler is set up prior to firing some code and then after said code has executed, if a certain time-threshhold
has passed, an alarm is raised dumping a stack trace to the console. I’m not entirely convinced I’m explaining this
appropriately so here’s some pseudo-code:</p>

<p><code type="python">
    def runloop():
        while True:
            signal.alarm(handler, 1)
            execute_next_block()
            if (time.time() - start) &lt; resolution:
                clear_signal() # Clear the signal if we're less than a second, otherwise it will alarm
</code></p>

<p>The blocking detection is a bit crude and can raise false positives if you have bits of code that churn
the CPU for longer than a second but it has been instrumental in incorporating <strong>non-blocking DNS</strong> support
into Eventlet, which was also introduced in 0.9.10 (ported over from Slide’s <a href="http://github.com/slideinc/gogreen">gogreen</a>
package).</p>

<p>If you are using Eventlet, I highly recommend running your code periodically with blocking detection enabled,
it is an invaluable tool for determining whether you’re running as fast and as asynchronous as possible. In my
case, it has been the difference between web services that are fast in development but slow under heavy stress,
and web services that are fast <strong>always</strong> regardless of load.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="software development" /><category term="python" /><summary type="html"><![CDATA[Colleagues of mine are all very familiar with my admiration of Eventlet, a Python concurrency library, built on top of greenlet, that provides lightweight “greenthreads” that naturally yield around I/O points. For me, the biggest draw of Eventlet besides its maturity, is how well it integrates with standard Python code. Any code that uses the built-in socket module can be “monkey-patched” (i.e. modified at runtime) to use the “green” version of the socket module which allows Eventlet to turn regular ol’ Python into code with asynchronous I/O. The problem with using libraries like Eventlet, is that some Python code just blocks, meaning that code will hit an I/O point and not yield but instead block the entire process until that network operation completes. In practical terms, imagine you have a web crawler that uses 10 “green threads”, each crawling a different site. The first greenthread (GT1) will send an HTTP request to the first site, then it will yield to GT2 and so on. If each HTTP request blocks for 100ms, that means when crawling the 10 sites, you’re going to block the whole process, preventing anything from running, for a whole second. Doesn’t sound too terrible, but imagine you’ve got 1000 greenthreads, instead of everything smoothly yielding from one thread to another the process will lock up very often resulting in painful slowdowns. Starting with Eventlet 0.9.10 “blocking detection” code has been incorporated into Eventlet to make it far easier for developers to find these portions of code that can block the entire process. import eventlet.debug eventlet.debug.hub_blocking_detection(True) While using the blocking detection is fairly simple, its implementation is a bit “magical” in that it’s not entirely obvious how it works. The detector is built around signals, inside of Eventlet a signal handler is set up prior to firing some code and then after said code has executed, if a certain time-threshhold has passed, an alarm is raised dumping a stack trace to the console. I’m not entirely convinced I’m explaining this appropriately so here’s some pseudo-code: def runloop(): while True: signal.alarm(handler, 1) execute_next_block() if (time.time() - start) &lt; resolution: clear_signal() # Clear the signal if we're less than a second, otherwise it will alarm The blocking detection is a bit crude and can raise false positives if you have bits of code that churn the CPU for longer than a second but it has been instrumental in incorporating non-blocking DNS support into Eventlet, which was also introduced in 0.9.10 (ported over from Slide’s gogreen package). If you are using Eventlet, I highly recommend running your code periodically with blocking detection enabled, it is an invaluable tool for determining whether you’re running as fast and as asynchronous as possible. In my case, it has been the difference between web services that are fast in development but slow under heavy stress, and web services that are fast always regardless of load.]]></summary></entry><entry><title type="html">Being a Libor, Addendum</title><link href="https://brokenco.de//2010/05/18/being-a-libor-addendum.html" rel="alternate" type="text/html" title="Being a Libor, Addendum" /><published>2010-05-18T00:00:00+00:00</published><updated>2010-05-18T00:00:00+00:00</updated><id>https://brokenco.de//2010/05/18/being-a-libor-addendum</id><content type="html" xml:base="https://brokenco.de//2010/05/18/being-a-libor-addendum.html"><![CDATA[<p>A couple of weeks ago I wrote a post on how to “<a href="http://unethicalblogger.com/posts/2010/04/be_libor">Be a Libor</a>”, trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to <a href="http://www.slide.com/corp/about-us.html">CTO at Slide</a>.</p>

<p>Over coffee today Libor offered up some finer points on the post in our discussion about building  teams. It is important, according to Libor, to maintain a “mental framework” within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).</p>

<p>Libor also had a fantastic quote from the conversation with regards to approaching new problems:</p>

<blockquote>
  <p>Just because there are multiple right answers, doesn’t mean there’s no wrong answers</p>
</blockquote>

<p>Depending on the complexity of the problems you’re facing there are likely a number of solutions but you still can get it wrong, particularly if you don’t remain consistent with your underlying mental framework for the project/organization.</p>

<p>As usual my discussions with Libor are interesting and enjoyable, he’s one of the most capable, thoughtful engineers I know, so I’m interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.</p>

<p><a href="http://icanhascheezburger.com/2007/05/13/god-speed-moon-cat/">God speed mooncat</a></p>]]></content><author><name>R. Tyler Croy</name></author><category term="slide" /><category term="opinion" /><category term="software development" /><category term="python" /><category term="apture" /><summary type="html"><![CDATA[A couple of weeks ago I wrote a post on how to “Be a Libor”, trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide. Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a “mental framework” within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!). Libor also had a fantastic quote from the conversation with regards to approaching new problems: Just because there are multiple right answers, doesn’t mean there’s no wrong answers Depending on the complexity of the problems you’re facing there are likely a number of solutions but you still can get it wrong, particularly if you don’t remain consistent with your underlying mental framework for the project/organization. As usual my discussions with Libor are interesting and enjoyable, he’s one of the most capable, thoughtful engineers I know, so I’m interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats. God speed mooncat]]></summary></entry><entry><title type="html">Is programming with Twisted really as awful as it sounds?</title><link href="https://brokenco.de//2010/05/12/is-programming-with-twisted-really-as-awful-as-it-sounds.html" rel="alternate" type="text/html" title="Is programming with Twisted really as awful as it sounds?" /><published>2010-05-12T00:00:00+00:00</published><updated>2010-05-12T00:00:00+00:00</updated><id>https://brokenco.de//2010/05/12/is-programming-with-twisted-really-as-awful-as-it-sounds</id><content type="html" xml:base="https://brokenco.de//2010/05/12/is-programming-with-twisted-really-as-awful-as-it-sounds.html"><![CDATA[<p>Early this week <a href="http://twitter.com/cansar">Can</a> forwarded <a href="http://www.quora.com/Is-programming-with-Twisted-really-as-awful-as-it-sounds">this post on Quora</a> to me, which asks the question:</p>

<blockquote>
  <p>Is programming with Twisted really as awful as it sounds?</p>
</blockquote>

<p>Yes. <em>Yes</em>. <strong>YES IT IS</strong>. <strong><em>HOLY CRAP IT’S AWFUL</em></strong></p>

<p>Here’s some good alternatives:</p>

<ul>
  <li><a href="http://eventlet.net">Eventlet</a>, my preference</li>
  <li><a href="http://gevent.org">gevent</a>, an alternative to Eventlet tied to libevent</li>
  <li><a href="http://www.java.com/">Java</a>. because let’s face it, if you’re using Twisted, you’ve already decided not to write Python, so use something with proper threading support.</li>
</ul>

<p>That is all.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="opinion" /><category term="python" /><summary type="html"><![CDATA[Early this week Can forwarded this post on Quora to me, which asks the question: Is programming with Twisted really as awful as it sounds? Yes. Yes. YES IT IS. HOLY CRAP IT’S AWFUL Here’s some good alternatives: Eventlet, my preference gevent, an alternative to Eventlet tied to libevent Java. because let’s face it, if you’re using Twisted, you’ve already decided not to write Python, so use something with proper threading support. That is all.]]></summary></entry><entry><title type="html">How-to: Using Avro with Eventlet</title><link href="https://brokenco.de//2010/05/07/how-to-using-avro-with-eventlet.html" rel="alternate" type="text/html" title="How-to: Using Avro with Eventlet" /><published>2010-05-07T00:00:00+00:00</published><updated>2010-05-07T00:00:00+00:00</updated><id>https://brokenco.de//2010/05/07/how-to-using-avro-with-eventlet</id><content type="html" xml:base="https://brokenco.de//2010/05/07/how-to-using-avro-with-eventlet.html"><![CDATA[<p>Working on the plumbing behind a sufficiently large web application I find
myself building services to meet my needs more often than not. Typically I
try to build single-purpose services, following in the unix philosophy, cobbling
together more complex tools based on a collection of distinct building blocks.
In order to connect these services a solid, fast and easy-to-use RPC library is
a requirement; enter <a href="http://hadoop.apache.org/avro/">Avro</a>.</p>

<hr />

<p><em>Note:</em> You can skip ahead and just start reading some source code by cloning my
<a href="http://github.com/rtyler/eventlet-avro-example">eventlet-avro-example</a> repository
from GitHub.</p>

<hr />

<p>Avro is part of the Hadoop project and has two primary components, data serialization
and RPC support. Some time ago I chose Avro for serializing all of <a id="aptureLink_LDwxZTTwKh" href="http://www.apture.com">Apture’s</a> metrics and logging
information, giving us a standardized framework for recording new events and processing
them after the fact. It was not until recently I started to take advantage of Avro’s
RPC support when building services with <a id="aptureLink_a4wlc7Bdkp" href="http://eventlet.net/doc/">Eventlet</a>. I’ve talked about Eventlet <a href="http://unethicalblogger.com/posts/2010/01/new_years_python_meme">before</a>, but
to recap:</p>

<blockquote>
  <p>Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it</p>
</blockquote>

<p>What this means in practice is that you can write highly concurrent network-based
services while keeping the code “synchronous” and easy to follow. Underneath
Eventlet is the “<a id="aptureLink_FICZSkfldQ" href="http://pypi.python.org/pypi/greenlet">greenlet</a>” library which implements coroutines for Python, which
allows Eventlet to switch between coroutines, or “green threads” whenever a network
call blocks.</p>

<p>Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building
their own transport layer for RPC calls, Avro sits on top of HTTP for its transport
layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use <a href="http://eventlet.net/doc/modules/wsgi.html">eventlet.wsgi</a> for building a fast, simple RPC server.
<!--break--></p>
<h3 id="defining-the-protocol">Defining the Protocol</h3>
<p>The first part of any Avro RPC project should be to define the protocol for RPC calls.
With Avro this entails a JSON-formatted specification, for our echo server example,
we have the following protocol:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{"protocol" : "AvroEcho",
"namespace" : "rpc.sample.echo",
"doc" : "Protocol for our AVRO echo server",
"types" : [],
"messages" : {
    "echo" : {
        "doc" : "Echo the string back",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    },
    "split" : {
        "doc" : "Split the string in two and echo",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    }
}}
</code></pre></div></div>

<p>The protocol can be deconstructed into two concrete portions, type definitions and
a message enumeration. For our echo server we don’t need any complex types, so the
<code class="language-plaintext highlighter-rouge">types</code> entry is empty. We do have two different messages defined, <code class="language-plaintext highlighter-rouge">echo</code> and <code class="language-plaintext highlighter-rouge">split</code>.
The message definition is a means of defining the actual remote-procedure-call,
services supporting this defined protocol will need to send responses for both kinds
of messages. For now, the messages are quite simple, they expect a <code class="language-plaintext highlighter-rouge">query</code> parameter
which should be a string, and are expected to return a string. Simple.</p>

<p>(This is defined in <a href="http://github.com/rtyler/eventlet-avro-example/blob/master/protocol.py">protocol.py</a> in the Git repo)</p>

<h3 id="implementing-a-client">Implementing a Client</h3>
<p>Implementing an Avro RPC client is simple, and the same whether you’re building a
service with Eventlet or any other Python library so I won’t dwell on the subject.
A client only needs to build two objects, an “HTTPTransceiver” which can be used
for multiple RPC calls and grafts additional logic on top of <code class="language-plaintext highlighter-rouge">httplib.HTTPConnection</code>
and a “Requestor”.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>client = avro.ipc.HTTPTransceiver(HOST, PORT)
requestor = avro.ipc.Requestor(protocol.EchoProtocol, client)
response = requestor.request('echo', {'query' : 'Hello World'})
</code></pre></div></div>

<p>You can also re-use for same <code class="language-plaintext highlighter-rouge">Requestor</code> object for multiple messages of the same
protocol. The three-line snippet above will send an RPC message <code class="language-plaintext highlighter-rouge">echo</code> to the server
and then return the response.</p>

<p>(This is elaborated more on in <a href="http://github.com/rtyler/eventlet-avro-example/blob/master/client.py">client.py</a> in the Git repo)</p>

<h3 id="building-the-server">Building the server</h3>
<p>Building the server to service these Avro RPC messages is the most complicated
piece of the puzzle, but it’s still remarkably simple. Inside the <code class="language-plaintext highlighter-rouge">server.py</code> you
will notice that we call <code class="language-plaintext highlighter-rouge">eventlet.monkey_patch()</code> at the top of the file. While not
strictly necessary inside the server since we’re relying on <code class="language-plaintext highlighter-rouge">eventlet.wsgi</code>for
writing to the socket. Regardless it’s a good habit to get into when working with
Eventlet, and would be required if our Avro-server was also an Avro-client, sending
requests to other services. Focusing on the simple use-case of returning responses
from the “echo” and “split” messages, first the WSGI server needs to be created:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>listener = eventlet.listen((HOST, PORT))
eventlet.wsgi.server(listener, wsgi_handler)
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">wsgi_handler</code> is a function which accepts the <code class="language-plaintext highlighter-rouge">environment</code> and <code class="language-plaintext highlighter-rouge">start_response</code>
arguments (per the WSGI “standard”). For the actually processing of the message,
you should refer to the <code class="language-plaintext highlighter-rouge">wsgi_handler</code> function in <code class="language-plaintext highlighter-rouge">server.py</code> in the example
repository.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def wsgi_handler(env, start_response):
    ## Only allow POSTs, which is what Avro should be doing
    if not env['REQUEST_METHOD'] == 'POST':
        start_response('500 Error', [('Content-Type', 'text/plain')])
        return ['Invalid REQUEST_METHOD\r\n']

    ## Pull the avro rpc message off of the POST data in `wsgi.input`
    reader = avro.ipc.FramedReader(env['wsgi.input'])
    request = reader.read_framed_message()
    response = responder.respond(request)

    ## avro.ipc.FramedWriter really wants a file-like object to write out to
    ## but since we're in WSGI-land we'll write to a StringIO and then output the
    ## buffer in a "proper" WSGI manner
    out = StringIO.StringIO()
    writer = avro.ipc.FramedWriter(out)
    writer.write_framed_message(response)

    start_response('200 OK', [('Content-Type', 'avro/binary')])
    return [out.getvalue()]
</code></pre></div></div>

<p>The only notable quirk with using Avro with a WSGI framework like
<code class="language-plaintext highlighter-rouge">eventlet.wsgi</code> is that some of Avro’s “writer” code expects to be given a raw
socket to write a response to, so we give it a <code class="language-plaintext highlighter-rouge">StringIO</code> object to write to and
return that buffer’s contents from <code class="language-plaintext highlighter-rouge">wsgi_handler</code>. The <code class="language-plaintext highlighter-rouge">wsgi_handler</code> function
above is “dumb” insofar that it’s simply passing the Avro request object into the
“responder” which is responsible for doing the work:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class EchoResponder(avro.ipc.Responder):
    def invoke(self, message, request):
        handler = 'handle_%s' % message.name
        if not hasattr(self, handler):
            raise Exception('I can\'t handle this message! (%s)' % message.name)
        return getattr(self, handler)(message, request)

    def handle_split(self, message, request):
        query = request['query']
        halfway = len(query) / 2
        return query[:halfway]

    def handle_echo(self, message, request):
        return request['query']
</code></pre></div></div>

<p>All in all, minus comments the server code is around 40 lines and fairly easy to
follow (refer to <a href="http://github.com/rtyler/eventlet-avro-example/blob/master/server.py">server.py</a> for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that.</p>

<p>If you’re curious about some of the other work I’ve been up to with Eventlet, <a href="http://github.com/rtyler">follow me on GitHub</a> :)</p>]]></content><author><name>R. Tyler Croy</name></author><category term="software development" /><category term="python" /><category term="apture" /><summary type="html"><![CDATA[Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro. Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub. Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture’s metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro’s RPC support when building services with Eventlet. I’ve talked about Eventlet before, but to recap: Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it What this means in practice is that you can write highly concurrent network-based services while keeping the code “synchronous” and easy to follow. Underneath Eventlet is the “greenlet” library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or “green threads” whenever a network call blocks. Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server. Defining the Protocol The first part of any Avro RPC project should be to define the protocol for RPC calls. With Avro this entails a JSON-formatted specification, for our echo server example, we have the following protocol: {"protocol" : "AvroEcho", "namespace" : "rpc.sample.echo", "doc" : "Protocol for our AVRO echo server", "types" : [], "messages" : { "echo" : { "doc" : "Echo the string back", "request" : [ {"name" : "query", "type" : "string"} ], "response" : "string", "errors" : ["string"] }, "split" : { "doc" : "Split the string in two and echo", "request" : [ {"name" : "query", "type" : "string"} ], "response" : "string", "errors" : ["string"] } }} The protocol can be deconstructed into two concrete portions, type definitions and a message enumeration. For our echo server we don’t need any complex types, so the types entry is empty. We do have two different messages defined, echo and split. The message definition is a means of defining the actual remote-procedure-call, services supporting this defined protocol will need to send responses for both kinds of messages. For now, the messages are quite simple, they expect a query parameter which should be a string, and are expected to return a string. Simple. (This is defined in protocol.py in the Git repo) Implementing a Client Implementing an Avro RPC client is simple, and the same whether you’re building a service with Eventlet or any other Python library so I won’t dwell on the subject. A client only needs to build two objects, an “HTTPTransceiver” which can be used for multiple RPC calls and grafts additional logic on top of httplib.HTTPConnection and a “Requestor”. client = avro.ipc.HTTPTransceiver(HOST, PORT) requestor = avro.ipc.Requestor(protocol.EchoProtocol, client) response = requestor.request('echo', {'query' : 'Hello World'}) You can also re-use for same Requestor object for multiple messages of the same protocol. The three-line snippet above will send an RPC message echo to the server and then return the response. (This is elaborated more on in client.py in the Git repo) Building the server Building the server to service these Avro RPC messages is the most complicated piece of the puzzle, but it’s still remarkably simple. Inside the server.py you will notice that we call eventlet.monkey_patch() at the top of the file. While not strictly necessary inside the server since we’re relying on eventlet.wsgifor writing to the socket. Regardless it’s a good habit to get into when working with Eventlet, and would be required if our Avro-server was also an Avro-client, sending requests to other services. Focusing on the simple use-case of returning responses from the “echo” and “split” messages, first the WSGI server needs to be created: listener = eventlet.listen((HOST, PORT)) eventlet.wsgi.server(listener, wsgi_handler) The wsgi_handler is a function which accepts the environment and start_response arguments (per the WSGI “standard”). For the actually processing of the message, you should refer to the wsgi_handler function in server.py in the example repository. def wsgi_handler(env, start_response): ## Only allow POSTs, which is what Avro should be doing if not env['REQUEST_METHOD'] == 'POST': start_response('500 Error', [('Content-Type', 'text/plain')]) return ['Invalid REQUEST_METHOD\r\n'] ## Pull the avro rpc message off of the POST data in `wsgi.input` reader = avro.ipc.FramedReader(env['wsgi.input']) request = reader.read_framed_message() response = responder.respond(request) ## avro.ipc.FramedWriter really wants a file-like object to write out to ## but since we're in WSGI-land we'll write to a StringIO and then output the ## buffer in a "proper" WSGI manner out = StringIO.StringIO() writer = avro.ipc.FramedWriter(out) writer.write_framed_message(response) start_response('200 OK', [('Content-Type', 'avro/binary')]) return [out.getvalue()] The only notable quirk with using Avro with a WSGI framework like eventlet.wsgi is that some of Avro’s “writer” code expects to be given a raw socket to write a response to, so we give it a StringIO object to write to and return that buffer’s contents from wsgi_handler. The wsgi_handler function above is “dumb” insofar that it’s simply passing the Avro request object into the “responder” which is responsible for doing the work: class EchoResponder(avro.ipc.Responder): def invoke(self, message, request): handler = 'handle_%s' % message.name if not hasattr(self, handler): raise Exception('I can\'t handle this message! (%s)' % message.name) return getattr(self, handler)(message, request) def handle_split(self, message, request): query = request['query'] halfway = len(query) / 2 return query[:halfway] def handle_echo(self, message, request): return request['query'] All in all, minus comments the server code is around 40 lines and fairly easy to follow (refer to server.py for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that. If you’re curious about some of the other work I’ve been up to with Eventlet, follow me on GitHub :)]]></summary></entry><entry><title type="html">Be a Libor</title><link href="https://brokenco.de//2010/04/30/be-a-libor.html" rel="alternate" type="text/html" title="Be a Libor" /><published>2010-04-30T00:00:00+00:00</published><updated>2010-04-30T00:00:00+00:00</updated><id>https://brokenco.de//2010/04/30/be-a-libor</id><content type="html" xml:base="https://brokenco.de//2010/04/30/be-a-libor.html"><![CDATA[<p>I reflect occasionally on how I’ve gotten to where I am right now, specifically to how I made the jump from “just some kid at a Piggly Wiggly in Texas” as <a id="aptureLink_7fpgpX6rLb" href="http://twitter.com/stuffonfire">Dave</a> once said, to the guy who knows <em>stuff</em> about <strong>things</strong>.  I often think about what pieces of the <a id="aptureLink_CJpdUZmrfu" href="http://twitter.com/slideinc">Slide</a> engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at <a id="aptureLink_jd3j6BSrUf" href="http://www.apture.com">Apture</a>.</p>

<p>The two pillars of engineering at Slide, at least in my naive world-view, were Dave and <a id="aptureLink_xrzzjPhkPZ" href="http://www.facebook.com/libor.michalek">Libor</a>. I joined Dave’s team when I joined Slide, and I left Libor’s team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave’s time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I’m at Apture, I’m trying to emulate Libor.</p>

<p>(<em>Note</em>: I do not intend to idolize either of them, but cite important characteristics)</p>

<p>To understand the Libor role, the phrase “the buck stops here” is useful. A Libor is the end of the line for engineering questions, unlike some organizations the “question-chain-of-command” is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, <em>you’re screwed</em>.</p>

<p>What does it take to be a Libor you may be thinking:
<!--break--></p>
<ul>
  <li>
    <p><strong>No Guessing:</strong>  When acting as a Libor, <em>knowing</em> is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.</p>
  </li>
  <li>
    <p><strong>Grok the stack:</strong>  A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that’s the Libor.</p>
  </li>
  <li>
    <p><strong>Maintenance and accountability:</strong> Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month’s and last year’s short-sighted decisions over and over. A Libor should not let himself nor colleagues “fire and forget” code, poor judgement will haunt a Libor for much longer than most people’s New Year’s resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.</p>
  </li>
  <li>
    <p><strong>Focus on Engineering:</strong>  Code quality and extendability are Libor’s primary focus, that is not to say that a Libor’s role is to impede product development, but rather ensure that it is properly framed. While a product manager’s primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn’t break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.</p>
  </li>
</ul>

<p>Playing the Libor character at Apture has been interesting to say the least, I’ve done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.</p>

<p>Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:</p>

<blockquote>
  <p>Tell me what you want, we’ll find a way to do it</p>
</blockquote>

<p><em>That’s</em> <a href="http://twitter.com/tristanharris/status/8355935929">a breakthrough</a>.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="slide" /><category term="opinion" /><category term="software development" /><category term="python" /><category term="apture" /><summary type="html"><![CDATA[I reflect occasionally on how I’ve gotten to where I am right now, specifically to how I made the jump from “just some kid at a Piggly Wiggly in Texas” as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture. The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave’s team when I joined Slide, and I left Libor’s team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave’s time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I’m at Apture, I’m trying to emulate Libor. (Note: I do not intend to idolize either of them, but cite important characteristics) To understand the Libor role, the phrase “the buck stops here” is useful. A Libor is the end of the line for engineering questions, unlike some organizations the “question-chain-of-command” is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you’re screwed. What does it take to be a Libor you may be thinking: No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over. Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that’s the Libor. Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month’s and last year’s short-sighted decisions over and over. A Libor should not let himself nor colleagues “fire and forget” code, poor judgement will haunt a Libor for much longer than most people’s New Year’s resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are. Focus on Engineering: Code quality and extendability are Libor’s primary focus, that is not to say that a Libor’s role is to impede product development, but rather ensure that it is properly framed. While a product manager’s primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn’t break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward. Playing the Libor character at Apture has been interesting to say the least, I’ve done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship. Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing: Tell me what you want, we’ll find a way to do it That’s a breakthrough.]]></summary></entry><entry><title type="html">Pyrage: Static isn’t just something on the radio</title><link href="https://brokenco.de//2010/02/26/pyrage-static-isnt-just-something-on-the-radio.html" rel="alternate" type="text/html" title="Pyrage: Static isn’t just something on the radio" /><published>2010-02-26T00:00:00+00:00</published><updated>2010-02-26T00:00:00+00:00</updated><id>https://brokenco.de//2010/02/26/pyrage-static-isnt-just-something-on-the-radio</id><content type="html" xml:base="https://brokenco.de//2010/02/26/pyrage-static-isnt-just-something-on-the-radio.html"><![CDATA[<p>Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I’m sure you’re thinking “But Dr. Tyler, Python is a <em>dynamic</em> language!”, it is indeed, but that does not mean there aren’t static variables.</p>

<p>The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you’re dealing with, it makes far more sense. Let’s take this static class variable for example:</p>

<p><code lang="python">&gt;&gt;&gt; class Foo(object):
...   my_list = []
...</code></p>
<blockquote>
  <blockquote>
    <blockquote>
      <p>f = Foo()
b = Foo()&lt;/code&gt;</p>
    </blockquote>
  </blockquote>
</blockquote>

<p>You’re trying to be clever, defining your class variables with their default variables outside of your <code class="language-plaintext highlighter-rouge">__init__</code> function, understandable, unless you ever intend on <strong>mutating</strong> that variable.
<code lang="python">&gt;&gt;&gt; f.my_list.append('O HAI')</code></p>
<blockquote>
  <blockquote>
    <blockquote>
      <p>print b.my_list
[‘O HAI’]
&lt;/code&gt;</p>
    </blockquote>
  </blockquote>
</blockquote>

<p>Still feeling clever? If that’s what you <em>wanted</em>, I bet you do, but if you wanted each class to have its own internal list you’ve inadvertantly introduced a bug where <em>any</em> and <em>every</em> time something mutates <code class="language-plaintext highlighter-rouge">my_list</code>, it will change for every single instance of <code class="language-plaintext highlighter-rouge">Foo</code>. The reason that this occurs is because <code class="language-plaintext highlighter-rouge">my_list</code> is tied to the class object <code class="language-plaintext highlighter-rouge">Foo</code> and not the <strong>instance</strong> of the <code class="language-plaintext highlighter-rouge">Foo</code> object (<code class="language-plaintext highlighter-rouge">f</code> or <code class="language-plaintext highlighter-rouge">b</code>). In effect <code class="language-plaintext highlighter-rouge">f.__class__.my_list</code> and <code class="language-plaintext highlighter-rouge">b.__class__.my_list</code> are the same object, in fact, the <code class="language-plaintext highlighter-rouge">__class__</code> objects of both those instances is the same as well. <code lang="python">&gt;&gt;&gt; id(f.__class__)
7680112</code></p>
<blockquote>
  <blockquote>
    <blockquote>
      <p>id(b.<strong>class</strong>)
7680112&lt;/code&gt;</p>
    </blockquote>
  </blockquote>
</blockquote>

<p><br clear="all" />
When using default/optional parameters for methods you can also run afoul of statics in Python, for example:<code lang="python">&gt;&gt;&gt; def somefunc(data=[]):
...    data.append(1)
...    print ('data', data)
...</code></p>
<blockquote>
  <blockquote>
    <blockquote>
      <p>somefunc()
(‘data’, [1])
somefunc()
(‘data’, [1, 1])
somefunc()
(‘data’, [1, 1, 1])
&lt;/code&gt;</p>
    </blockquote>
  </blockquote>
</blockquote>

<p>This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you’re adding the variable <code class="language-plaintext highlighter-rouge">data</code> to the <code class="language-plaintext highlighter-rouge">somefunc.func_defaults</code> tuple, which is being mutated when the function is being called. Bad programmer!</p>

<p>It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they’re difficult to spot, frustrating to deal with (“who the hell is changing my variable!”) and most importantly, easily prevented with a little understanding of how Python scoping works.</p>

<p>PYRAGE!
<!--break--></p>]]></content><author><name>R. Tyler Croy</name></author><category term="opinion" /><category term="software development" /><category term="python" /><summary type="html"><![CDATA[Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I’m sure you’re thinking “But Dr. Tyler, Python is a dynamic language!”, it is indeed, but that does not mean there aren’t static variables. The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you’re dealing with, it makes far more sense. Let’s take this static class variable for example: &gt;&gt;&gt; class Foo(object): ... my_list = [] ... f = Foo() b = Foo()&lt;/code&gt; You’re trying to be clever, defining your class variables with their default variables outside of your __init__ function, understandable, unless you ever intend on mutating that variable. &gt;&gt;&gt; f.my_list.append('O HAI') print b.my_list [‘O HAI’] &lt;/code&gt; Still feeling clever? If that’s what you wanted, I bet you do, but if you wanted each class to have its own internal list you’ve inadvertantly introduced a bug where any and every time something mutates my_list, it will change for every single instance of Foo. The reason that this occurs is because my_list is tied to the class object Foo and not the instance of the Foo object (f or b). In effect f.__class__.my_list and b.__class__.my_list are the same object, in fact, the __class__ objects of both those instances is the same as well. &gt;&gt;&gt; id(f.__class__) 7680112 id(b.class) 7680112&lt;/code&gt; When using default/optional parameters for methods you can also run afoul of statics in Python, for example:&gt;&gt;&gt; def somefunc(data=[]): ... data.append(1) ... print ('data', data) ... somefunc() (‘data’, [1]) somefunc() (‘data’, [1, 1]) somefunc() (‘data’, [1, 1, 1]) &lt;/code&gt; This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you’re adding the variable data to the somefunc.func_defaults tuple, which is being mutated when the function is being called. Bad programmer! It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they’re difficult to spot, frustrating to deal with (“who the hell is changing my variable!”) and most importantly, easily prevented with a little understanding of how Python scoping works. PYRAGE!]]></summary></entry><entry><title type="html">If you want a viral license, use the GPL</title><link href="https://brokenco.de//2010/02/23/if-you-want-a-viral-license-use-the-gpl.html" rel="alternate" type="text/html" title="If you want a viral license, use the GPL" /><published>2010-02-23T00:00:00+00:00</published><updated>2010-02-23T00:00:00+00:00</updated><id>https://brokenco.de//2010/02/23/if-you-want-a-viral-license-use-the-gpl</id><content type="html" xml:base="https://brokenco.de//2010/02/23/if-you-want-a-viral-license-use-the-gpl.html"><![CDATA[<p>My “roots” in the open source community come from the BSD side of the open source spectrum, my first major introduction being involvement with <a id="aptureLink_Z6pelwUEYA" href="http://en.wikipedia.org/wiki/FreeBSD">FreeBSD</a> and <a id="aptureLink_ZXZxVq5WFh" href="http://en.wikipedia.org/wiki/OpenBSD">OpenBSD</a>. It is not surprising that my licensing preferences fall on the BSD (2 or 3 clause) or MIT licenses, the MIT license reading as follows:&lt;blockquote&gt;&lt;p&gt;Copyright (c) [year] [copyright holders]</p>
<p>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
<p>
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
<p>
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.&lt;/blockquote&gt;

I bring the subject up because I wanted to address a brief "kerfuffle" that occurred recently on the <a id="aptureLink_0mMM3DzSHh" href="http://eventlet.net/">Eventlet</a> mailing list with the maintainer of <a id="aptureLink_BWth7wZxHe" href="http://www.gevent.org/">gevent</a>, a fork/rewrite of Eventlet. Both projects are MIT licensed which gives anybody that would like to fork the source code of either project a great deal of leeway to hack about with the code, commercialize it, etc.
<!--break-->
**Disclaimer**: I personally am a fan of Eventlet, use it quite often, and have recently taking up maintaining Spawning, a WSGI server that supports multiple processes/threads, non-blocking I/O and graceful code reloading, built on top of Eventlet.

The "kerfuffle" occurred after Ryan, the maintainer of Eventlet, took a few good modules from gevent; *shocking* as it may seem, a developer working with liberally licensed code took liberally licensed code from a similar project. The issue that the maintainer of gevent took with the incorporation of his code was all about attribution:

&gt; I don't mind you borrowing code from gevent, the license permits it. However, please make it clear where are you getting the code from.

Upon first reading the email, I doubled over to the [eventlet source on Bitbucket](http://bitbucket.org/which_linden/eventlet/), checked the files that were incorporated into the codebase ([timeout.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/timeout.py) and [queue.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/queue.py))and sure enough the copyright attributing the original author were still in tact, surely this is a non-issue?

Unfortunately not, license pedantry is an open source community past-time, right up their with drinking and shouting. When I replied mentioning that the copyrights were correctly in place, mentioning that both projects were MIT licensed so both constraints of the license were met, that is, the MIT license notice was included with the code. In essence the disagreement revolves around what the phrase "this permission notice shall be included" entail, my interpretation of the license is such that the MIT license itself shall be included, not the specific file with additions from one project to another; after sending off my mail, I received the following reply:

&gt; Ok, it's acceptable to use one LICENSE file but only if the copyright notice from gevent is present unchanged.
&gt; 
&gt; That is, take the notice from here http://bitbucket.org/denis/gevent/src/tip/LICENSE (the line with the url), and put it into eventlet's LICENSE, on a separate line. (It's OK to add "Copyright (c) 2009-2010" before it to make it in line with others).
&gt; 
&gt; That would settle it.

Slightly pedantic in my opinion, the MIT license enumerates a line for copyright holders which has been hijacked for other information that the maintainer of gevent would like to propagate, I don't necessarily agree, but this is a mailing list not a court of law, so I'll allow it. The thread continues:

&gt; The license did not change. I've only updated the copyright notice to include the url of the project to protect against abusive borrowers, that's it.

This is where I draw the line, go all in, plant my flag in the sand and other unnecessary metaphors. **Abusive borrowers?** Analyzing the semantics of the phrase alone makes my head hurt, I have a mental image of two old ladies wherein one says to the other: "may I borrow a cup of sugar, you horse-faced hunch-backed bucket of moron?" The rest of the email is full of similarly head-hurting quotes, for brevity I won't include them here (you can read the thread [in the archives](https://lists.secondlife.com/pipermail/eventletdev/2010-February/000731.html)).

I'm simply dumbfounded by the ignorance of what the MIT license actually *means*, unlike the LGPL or the GPL license which were specifically drafted to protect against "abusive borrowers", <a id="aptureLink_grMTA0vhuq" href="http://lwn.net/Articles/51570/">such as Cisco</a>, the MIT license is so open it's *almost* public domain. 

To a certain extent I can understand the emotions behind the thread on the mailing list, I don't agree with them. If you're seeking attribution past the copyright line in a header, than perhaps the "original" [4 clause BSD license](http://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_.28original_.22BSD_License.22.29) is for you, or perhaps the LGPL or GPL which give you more control over what happens to the source code you originate? If this is what you're after, the MIT license is the wrong license.
</p></p></p>]]></content><author><name>R. Tyler Croy</name></author><category term="opinion" /><category term="python" /><summary type="html"><![CDATA[My “roots” in the open source community come from the BSD side of the open source spectrum, my first major introduction being involvement with FreeBSD and OpenBSD. It is not surprising that my licensing preferences fall on the BSD (2 or 3 clause) or MIT licenses, the MIT license reading as follows:&lt;blockquote&gt;&lt;p&gt;Copyright (c) [year] [copyright holders] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.&lt;/blockquote&gt; I bring the subject up because I wanted to address a brief "kerfuffle" that occurred recently on the Eventlet mailing list with the maintainer of gevent, a fork/rewrite of Eventlet. Both projects are MIT licensed which gives anybody that would like to fork the source code of either project a great deal of leeway to hack about with the code, commercialize it, etc. **Disclaimer**: I personally am a fan of Eventlet, use it quite often, and have recently taking up maintaining Spawning, a WSGI server that supports multiple processes/threads, non-blocking I/O and graceful code reloading, built on top of Eventlet. The "kerfuffle" occurred after Ryan, the maintainer of Eventlet, took a few good modules from gevent; *shocking* as it may seem, a developer working with liberally licensed code took liberally licensed code from a similar project. The issue that the maintainer of gevent took with the incorporation of his code was all about attribution: &gt; I don't mind you borrowing code from gevent, the license permits it. However, please make it clear where are you getting the code from. Upon first reading the email, I doubled over to the [eventlet source on Bitbucket](http://bitbucket.org/which_linden/eventlet/), checked the files that were incorporated into the codebase ([timeout.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/timeout.py) and [queue.py](http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/queue.py))and sure enough the copyright attributing the original author were still in tact, surely this is a non-issue? Unfortunately not, license pedantry is an open source community past-time, right up their with drinking and shouting. When I replied mentioning that the copyrights were correctly in place, mentioning that both projects were MIT licensed so both constraints of the license were met, that is, the MIT license notice was included with the code. In essence the disagreement revolves around what the phrase "this permission notice shall be included" entail, my interpretation of the license is such that the MIT license itself shall be included, not the specific file with additions from one project to another; after sending off my mail, I received the following reply: &gt; Ok, it's acceptable to use one LICENSE file but only if the copyright notice from gevent is present unchanged. &gt; &gt; That is, take the notice from here http://bitbucket.org/denis/gevent/src/tip/LICENSE (the line with the url), and put it into eventlet's LICENSE, on a separate line. (It's OK to add "Copyright (c) 2009-2010" before it to make it in line with others). &gt; &gt; That would settle it. Slightly pedantic in my opinion, the MIT license enumerates a line for copyright holders which has been hijacked for other information that the maintainer of gevent would like to propagate, I don't necessarily agree, but this is a mailing list not a court of law, so I'll allow it. The thread continues: &gt; The license did not change. I've only updated the copyright notice to include the url of the project to protect against abusive borrowers, that's it. This is where I draw the line, go all in, plant my flag in the sand and other unnecessary metaphors. **Abusive borrowers?** Analyzing the semantics of the phrase alone makes my head hurt, I have a mental image of two old ladies wherein one says to the other: "may I borrow a cup of sugar, you horse-faced hunch-backed bucket of moron?" The rest of the email is full of similarly head-hurting quotes, for brevity I won't include them here (you can read the thread [in the archives](https://lists.secondlife.com/pipermail/eventletdev/2010-February/000731.html)). I'm simply dumbfounded by the ignorance of what the MIT license actually *means*, unlike the LGPL or the GPL license which were specifically drafted to protect against "abusive borrowers", such as Cisco, the MIT license is so open it's *almost* public domain. To a certain extent I can understand the emotions behind the thread on the mailing list, I don't agree with them. If you're seeking attribution past the copyright line in a header, than perhaps the "original" [4 clause BSD license](http://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_.28original_.22BSD_License.22.29) is for you, or perhaps the LGPL or GPL which give you more control over what happens to the source code you originate? If this is what you're after, the MIT license is the wrong license.]]></summary></entry><entry><title type="html">Supporting Python 3 is a Ghetto</title><link href="https://brokenco.de//2010/02/21/supporting-python-3-is-a-ghetto.html" rel="alternate" type="text/html" title="Supporting Python 3 is a Ghetto" /><published>2010-02-21T00:00:00+00:00</published><updated>2010-02-21T00:00:00+00:00</updated><id>https://brokenco.de//2010/02/21/supporting-python-3-is-a-ghetto</id><content type="html" xml:base="https://brokenco.de//2010/02/21/supporting-python-3-is-a-ghetto.html"><![CDATA[<p>In my spurious free time I maintain a few Python modules (<a id="aptureLink_LvMqViext1" href="http://github.com/rtyler/py-yajl">py-yajl</a>, <a id="aptureLink_SEruJN7rBc" href="http://en.wikipedia.org/wiki/CheetahTemplate">Cheetah</a>, <a id="aptureLink_3HQW6OMHEx" href="http://github.com/rtyler/PyECC">PyECC</a>) and am semi-involved in a couple others (<a id="aptureLink_1I31I3RdtY" href="http://www.djangoproject.com/">Django</a>, <a id="aptureLink_7qs5LoY2eY" href="http://eventlet.net/">Eventlet</a>), only one of which properly supports Python 3. For the uninitiated, Python 3 is a backwards incompatible progression of the Python language and CPython implementation thereof, it’s represented significant challenges for the Python community insofar that supporting Python 2.xx, which is in wide deployment, and Python 3.xx simultaneously is difficult.</p>

<p>As it stands now my primary development environment is Python 2.6 on Linux/amd64, which means I get to take advantage of some of the nice things that were added to Python 3 and then back-ported to Python 2.6/2.7. Regular readers know about my undying love for Hudson, a Java-based continuous integration server, which I use to test and build all of the Python projects that I work on. While working this weekend I noticed that one of my C-based projects (py-yajl) was failing to link properly on Python 2.4 and 2.5. It might be easy to cut-off support for Python 2.4, which was first released over <strong>four years</strong> ago, there are still a number of heavy users of 2.4 (such as <a id="aptureLink_k20Tw96O5B" href="http://www.crunchbase.com/company/slide">Slide</a>), in fact it’s still the default <code class="language-plaintext highlighter-rouge">/usr/bin/python</code> on Red Hat Enterprise Linux 5. What makes this C-based module special, is that thanks to <a id="aptureLink_l6Vcy3ytZB" href="http://twitter.com/teepark">Travis</a>, it runs properly on Python 3.1 as well. Since the Python C-API has been <em>fairly</em> stable through the 2 series into Python 3, maintaining a C-based module that supports multiple versions of Python.</p>

<p>In this case, it’s as easy as some simple pre-processor definitions:<code lang="c">#if PY_MAJOR_VERSION &gt;= 3
#define IS_PYTHON3
#endif</code>Which I can use further down the line to modify the handling some of the minor internal changes for Python 3:<code lang="c">#ifdef IS_PYTHON3
    result = _internal_decode((_YajlDecoder *)decoder, PyBytes_AsString(bufferstring),
                PyBytes_Size(bufferstring));
    Py_XDECREF(bufferstring);
#else
    result = _internal_decode((_YajlDecoder *)decoder, PyString_AsString(buffer),
                  PyString_Size(buffer));
#endif </code></p>

<p>Not particularly <em>pretty</em> but it gets the job done, supporting all major versions of Python.</p>

<h3 id="python-on-python">Python on Python</h3>
<p>Writing modules in C is fun, can give you pretty good performance, but is not something you would want to do with a <strong>large</strong> package like Django (for example). Python is the language we all know and love to work with, a much more pleasant language to work with than C. If you build packages in pure Python, those packages have a much better chance running on top of IronPython or Jython, and the entire Python ecosystem is better for it.</p>

<p>A few weeks ago when I started to look deeper into the possibility of Cheetah support for Python 3, I found a process riddled with faults. First a disclaimer, Cheetah is almost <strong>ten years</strong> old; it’s one of the oldest Python projects I can think of that’s still chugging along. This translates into some <em>very</em> old looking code, most people who are new to the language aren’t familiar with some of the ways the language has changed in the past five years, let alone ten.</p>

<p>The current means of supporting Python 3 with pure Python packages is as follows:</p>

<ol>
  <li>Refactor the code enough such that <code class="language-plaintext highlighter-rouge">2to3</code> can process it</li>
  <li>Run <a id="aptureLink_GtN83eZUU3" href="http://docs.python.org/library/2to3.html">2to3</a> over the codebase, with the <code class="language-plaintext highlighter-rouge">-w</code> option to literally write the changes to the files</li>
  <li>Test your code on Python 3 (if it fails, go back to step 1)</li>
  <li>Create a source tarball, post to <a id="aptureLink_lvET3CCrpS" href="http://pypi.python.org/">PyPI</a>, continue developing in Python 2.xx</li>
</ol>

<p>I’m hoping you spotted the same problem with this model that I did, due to the reliance on <code class="language-plaintext highlighter-rouge">2to3</code> you are now trapped into <strong>always</strong> developing Python targeting Python <strong>2</strong>. This model will never succeed in moving people to Python 3, regardless of what amazing improvements it contains (such as the Unladen Swallow work) because you cannot develop on a day-to-day basis with Python 3, it’s a magic conversion tool away.</p>

<p>Unlike with a C module for Python, I cannot <code class="language-plaintext highlighter-rouge">#ifdef</code> certain segments of code in and out, which forces me to constantly use <code class="language-plaintext highlighter-rouge">2to3</code> <em>or</em> fork my code and maintain two separate branches of my project, duplicating the work for every change. With Python 2 sticking around on the scene for years to come (I don;t believe 2.7 will be the last release) I cannot imagine <strong>either</strong> of these workflows making sense long term.</p>

<p>At a fundamental level, supporting Python 3 does not make sense for anybody developing modules, particularly open source ones. Despite Python 3 being “the future”, it is currently impossible to develop using Python 3, maintaining support for Python 2, which <strong>all</strong> of us have to do. With enterprise operating systems like <a id="aptureLink_ehh7mOge8i" href="http://www.crunchbase.com/product/red-hat-enterprise-linux">Red Hat</a> or <a id="aptureLink_CklLBYgoAK" href="http://www.novell.com/linux/">SuSE</a> only now starting to get on board with Python 2.5 and Python 2.6, you can be certain that we’re more than five years away from seeing Python 3 installed by default on any production machines.
<!--break--></p>]]></content><author><name>R. Tyler Croy</name></author><category term="software development" /><category term="cheetah" /><category term="python" /><summary type="html"><![CDATA[In my spurious free time I maintain a few Python modules (py-yajl, Cheetah, PyECC) and am semi-involved in a couple others (Django, Eventlet), only one of which properly supports Python 3. For the uninitiated, Python 3 is a backwards incompatible progression of the Python language and CPython implementation thereof, it’s represented significant challenges for the Python community insofar that supporting Python 2.xx, which is in wide deployment, and Python 3.xx simultaneously is difficult. As it stands now my primary development environment is Python 2.6 on Linux/amd64, which means I get to take advantage of some of the nice things that were added to Python 3 and then back-ported to Python 2.6/2.7. Regular readers know about my undying love for Hudson, a Java-based continuous integration server, which I use to test and build all of the Python projects that I work on. While working this weekend I noticed that one of my C-based projects (py-yajl) was failing to link properly on Python 2.4 and 2.5. It might be easy to cut-off support for Python 2.4, which was first released over four years ago, there are still a number of heavy users of 2.4 (such as Slide), in fact it’s still the default /usr/bin/python on Red Hat Enterprise Linux 5. What makes this C-based module special, is that thanks to Travis, it runs properly on Python 3.1 as well. Since the Python C-API has been fairly stable through the 2 series into Python 3, maintaining a C-based module that supports multiple versions of Python. In this case, it’s as easy as some simple pre-processor definitions:#if PY_MAJOR_VERSION &gt;= 3 #define IS_PYTHON3 #endifWhich I can use further down the line to modify the handling some of the minor internal changes for Python 3:#ifdef IS_PYTHON3 result = _internal_decode((_YajlDecoder *)decoder, PyBytes_AsString(bufferstring), PyBytes_Size(bufferstring)); Py_XDECREF(bufferstring); #else result = _internal_decode((_YajlDecoder *)decoder, PyString_AsString(buffer), PyString_Size(buffer)); #endif Not particularly pretty but it gets the job done, supporting all major versions of Python. Python on Python Writing modules in C is fun, can give you pretty good performance, but is not something you would want to do with a large package like Django (for example). Python is the language we all know and love to work with, a much more pleasant language to work with than C. If you build packages in pure Python, those packages have a much better chance running on top of IronPython or Jython, and the entire Python ecosystem is better for it. A few weeks ago when I started to look deeper into the possibility of Cheetah support for Python 3, I found a process riddled with faults. First a disclaimer, Cheetah is almost ten years old; it’s one of the oldest Python projects I can think of that’s still chugging along. This translates into some very old looking code, most people who are new to the language aren’t familiar with some of the ways the language has changed in the past five years, let alone ten. The current means of supporting Python 3 with pure Python packages is as follows: Refactor the code enough such that 2to3 can process it Run 2to3 over the codebase, with the -w option to literally write the changes to the files Test your code on Python 3 (if it fails, go back to step 1) Create a source tarball, post to PyPI, continue developing in Python 2.xx I’m hoping you spotted the same problem with this model that I did, due to the reliance on 2to3 you are now trapped into always developing Python targeting Python 2. This model will never succeed in moving people to Python 3, regardless of what amazing improvements it contains (such as the Unladen Swallow work) because you cannot develop on a day-to-day basis with Python 3, it’s a magic conversion tool away. Unlike with a C module for Python, I cannot #ifdef certain segments of code in and out, which forces me to constantly use 2to3 or fork my code and maintain two separate branches of my project, duplicating the work for every change. With Python 2 sticking around on the scene for years to come (I don;t believe 2.7 will be the last release) I cannot imagine either of these workflows making sense long term. At a fundamental level, supporting Python 3 does not make sense for anybody developing modules, particularly open source ones. Despite Python 3 being “the future”, it is currently impossible to develop using Python 3, maintaining support for Python 2, which all of us have to do. With enterprise operating systems like Red Hat or SuSE only now starting to get on board with Python 2.5 and Python 2.6, you can be certain that we’re more than five years away from seeing Python 3 installed by default on any production machines.]]></summary></entry><entry><title type="html">Building a game for ET. Day 1 with Pygame</title><link href="https://brokenco.de//2010/02/14/building-a-game-for-et-day-1-with-pygame.html" rel="alternate" type="text/html" title="Building a game for ET. Day 1 with Pygame" /><published>2010-02-14T00:00:00+00:00</published><updated>2010-02-14T00:00:00+00:00</updated><id>https://brokenco.de//2010/02/14/building-a-game-for-et-day-1-with-pygame</id><content type="html" xml:base="https://brokenco.de//2010/02/14/building-a-game-for-et-day-1-with-pygame.html"><![CDATA[<p>Earlier this week I was checking out <a id="aptureLink_8JIIdIavnW" href="http://en.wikipedia.org/wiki/Pygame">Pygame</a>, pondering what I could possibly build with it that could keep me motivated enough to finish it. Motivation would like be the primary problem for me with any amount of game programming; I’m not a gamer, I don’t harbor a dislike of games, they’re just not something I typically spend time playing (I do like to play “haggard late night open-source hacker” though, that’s a fun one). Friday night I stumbled across an idea, ET likes to play (casual) games, perhaps we could write a game together; ask any engineer at EA or Ubisoft, there’s nothing more romantic than working on a game.</p>

<p>Talking over the idea with ET on the ride home from the office, we talked about creating a typing-oriented game and started to brainstorm. The tricky aspect of a typing-oriented game is you have to walk the fine line of “educational gaming”, that is to say, the game’s goal is <strong>not</strong> to teach the player how to type. That sucks. Contrasted to some other games where the means of progressing in some games is by solving a puzzle, killing noobs in others, in this game we wanted the player to progress through levels/situations with their typing ability (ET finds this fun, we do not have this in common).</p>

<p>Over pizza we discussed more about how the levels would work, I decided that I wanted to use stories/articles instead of random words for the “content” of the game. We settled on a couple fundamental concepts: the player would earn coins by correctly completing a words as the scrolled from right to left (similar to a ticker tape), they would lose coins if they made a mistake or could not keep up. After a player completed a level (i.e. a “story”) they would find themselves in a “store” of sorts, where they could purchase “tools” for future levels with their coins. The tools we decided would be a <em>very</em> important, as the player reached their upper bound of typing speed the utility of these various tools would necessary as a means of strategically conquering the level. One of the few things we didn’t particularly cover was the “end game”, whether the player would simply play increasingly more difficult levels (a la <a id="aptureLink_S4eypmdIVi" href="http://en.wikipedia.org/wiki/Tetris">Tetris</a>) or if they could actually “beat” the game. With at least the basics of the concept sketched out, it was time to start writing <em>some</em> code.</p>

<h3 id="starting-with-pygame">Starting with Pygame</h3>
<p>It’s <strong>incredibly</strong> important to mention that I’ve <em>never</em> programmed a game before. <em>Never-ever</em>. From my work with network programming I was already familiar with the concept of the run-loop that’s pretty core to Pygame, but I had never really made use of any to animate objects on the screen or deal with handling any kind of events from mouse movements to key presses, etc. Fortunately I’m already a professional Python developer, so writing code wasn’t the difficult part so much as laying it out. Orienting things into classes to handle separate components such as animating text (which is a painful in Pygame, in my opinion) to keeping track of user-input.</p>

<p>Animating text across the screen wasn’t particularly difficult, with Pygame you first create your primary “surface” (i.e. the window) and then you can render things onto that surface. With text, you end up rendering a surface which contains your text, “hello world” which you then place onto the primary surface. Easy peasy thus far:
<code type="python">
    import pygame
    surface = pygame.display.set_mode((640, 680), pygame.HWSURFACE | pygame.DOUBLEBUF)
    font = pygame.font.SysFont('Courier', 42)
    ### render(text, antialias, rgb color tuple
    font_surface = font.render('hello world', 0, (0, 0, 0)) 
    ### draw `font_surface` onto `surface` at (x=0, y=0)
    start_x, start_y = 0, 0
    surface.blit(font_surface, (start_x, start_y))
    while True:
         ### Holy runloop batman
         pygame.display.update()
</code></p>

<p>That was fun, I now have “hello world” rendered onto my screen, now to animate I suppose I’ll just render <code class="language-plaintext highlighter-rouge">font_surface</code> a little further right every iteration of the runloop, i.e.
<code type="python">
    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         start_x += 0.5
         pygame.display.update()
</code>
This blurs the text however, so I then changed to:
<code type="python">
    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         surface.fill((0, 0, 0))
         start_x += 0.5
         pygame.display.update()
</code>
This will cause the (primary) surface to be repainted (washed over) every iteration of the runloop ensuring that the text will properly animate, drawing the text in one spot, wiping the surface then drawing it slightly further to the left, resulting in the scrolling animation. All’s fine and good until you determine that you want to have <em>other</em> elements on the screen and you also don’t want to redraw them every time around the carousel. I then discovered how to “fill” just one particular rectangle on the surface, i.e. the rectangle behind the text:
<code type="python">
    text_w, text_h = font.size('hello world')
    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         surface.fill((0, 0, 0), rect=pygame.Rect(start_x, start_y, text_w, text_h))
         start_x += 0.5
         pygame.display.update()
</code></p>

<p>Once I was able to get text properly scrolling across the screen, the rest of the afternoon of hacking was far easier. My confidence in my ability to grok Pygame in order to do what I wanted. I then set forth organizing my code into some logical classes, for example I created a <code class="language-plaintext highlighter-rouge">LetterSpool</code> class which would record the user’s progress through the current word, rendering it at the bottom-center of the screen and firing an event when the user hit the space bar (denoting the word “complete”), additionally I wrapped my text animation code into <code class="language-plaintext highlighter-rouge">AnimatedWord</code> so I could easily string words together to scroll across the screen in conjunction similar to a textual screensaver.</p>

<p>Not a whole lot more to write about with regards to my progress today, hooked up some music and basic sound effects which was trivial after looking at some sample code. Next I need to start addressing some more fundamentals for user-interaction: scoring and level-changing.</p>

<p>You can track the progress of the game “Typy” (pronounced: <code class="language-plaintext highlighter-rouge">typey</code>) <a href="http://github.com/rtyler/typy">on GitHub</a></p>

<center><img src="http://agentdero.cachefly.net/scratch/typy_day1.png" /></center>]]></content><author><name>R. Tyler Croy</name></author><category term="python" /><summary type="html"><![CDATA[Earlier this week I was checking out Pygame, pondering what I could possibly build with it that could keep me motivated enough to finish it. Motivation would like be the primary problem for me with any amount of game programming; I’m not a gamer, I don’t harbor a dislike of games, they’re just not something I typically spend time playing (I do like to play “haggard late night open-source hacker” though, that’s a fun one). Friday night I stumbled across an idea, ET likes to play (casual) games, perhaps we could write a game together; ask any engineer at EA or Ubisoft, there’s nothing more romantic than working on a game. Talking over the idea with ET on the ride home from the office, we talked about creating a typing-oriented game and started to brainstorm. The tricky aspect of a typing-oriented game is you have to walk the fine line of “educational gaming”, that is to say, the game’s goal is not to teach the player how to type. That sucks. Contrasted to some other games where the means of progressing in some games is by solving a puzzle, killing noobs in others, in this game we wanted the player to progress through levels/situations with their typing ability (ET finds this fun, we do not have this in common). Over pizza we discussed more about how the levels would work, I decided that I wanted to use stories/articles instead of random words for the “content” of the game. We settled on a couple fundamental concepts: the player would earn coins by correctly completing a words as the scrolled from right to left (similar to a ticker tape), they would lose coins if they made a mistake or could not keep up. After a player completed a level (i.e. a “story”) they would find themselves in a “store” of sorts, where they could purchase “tools” for future levels with their coins. The tools we decided would be a very important, as the player reached their upper bound of typing speed the utility of these various tools would necessary as a means of strategically conquering the level. One of the few things we didn’t particularly cover was the “end game”, whether the player would simply play increasingly more difficult levels (a la Tetris) or if they could actually “beat” the game. With at least the basics of the concept sketched out, it was time to start writing some code. Starting with Pygame It’s incredibly important to mention that I’ve never programmed a game before. Never-ever. From my work with network programming I was already familiar with the concept of the run-loop that’s pretty core to Pygame, but I had never really made use of any to animate objects on the screen or deal with handling any kind of events from mouse movements to key presses, etc. Fortunately I’m already a professional Python developer, so writing code wasn’t the difficult part so much as laying it out. Orienting things into classes to handle separate components such as animating text (which is a painful in Pygame, in my opinion) to keeping track of user-input. Animating text across the screen wasn’t particularly difficult, with Pygame you first create your primary “surface” (i.e. the window) and then you can render things onto that surface. With text, you end up rendering a surface which contains your text, “hello world” which you then place onto the primary surface. Easy peasy thus far: import pygame surface = pygame.display.set_mode((640, 680), pygame.HWSURFACE | pygame.DOUBLEBUF) font = pygame.font.SysFont('Courier', 42) ### render(text, antialias, rgb color tuple font_surface = font.render('hello world', 0, (0, 0, 0)) ### draw `font_surface` onto `surface` at (x=0, y=0) start_x, start_y = 0, 0 surface.blit(font_surface, (start_x, start_y)) while True: ### Holy runloop batman pygame.display.update() That was fun, I now have “hello world” rendered onto my screen, now to animate I suppose I’ll just render font_surface a little further right every iteration of the runloop, i.e. while True: ### Holy runloop batman surface.blit(font_surface, (start_x, start_y)) start_x += 0.5 pygame.display.update() This blurs the text however, so I then changed to: while True: ### Holy runloop batman surface.blit(font_surface, (start_x, start_y)) surface.fill((0, 0, 0)) start_x += 0.5 pygame.display.update() This will cause the (primary) surface to be repainted (washed over) every iteration of the runloop ensuring that the text will properly animate, drawing the text in one spot, wiping the surface then drawing it slightly further to the left, resulting in the scrolling animation. All’s fine and good until you determine that you want to have other elements on the screen and you also don’t want to redraw them every time around the carousel. I then discovered how to “fill” just one particular rectangle on the surface, i.e. the rectangle behind the text: text_w, text_h = font.size('hello world') while True: ### Holy runloop batman surface.blit(font_surface, (start_x, start_y)) surface.fill((0, 0, 0), rect=pygame.Rect(start_x, start_y, text_w, text_h)) start_x += 0.5 pygame.display.update() Once I was able to get text properly scrolling across the screen, the rest of the afternoon of hacking was far easier. My confidence in my ability to grok Pygame in order to do what I wanted. I then set forth organizing my code into some logical classes, for example I created a LetterSpool class which would record the user’s progress through the current word, rendering it at the bottom-center of the screen and firing an event when the user hit the space bar (denoting the word “complete”), additionally I wrapped my text animation code into AnimatedWord so I could easily string words together to scroll across the screen in conjunction similar to a textual screensaver. Not a whole lot more to write about with regards to my progress today, hooked up some music and basic sound effects which was trivial after looking at some sample code. Next I need to start addressing some more fundamentals for user-interaction: scoring and level-changing. You can track the progress of the game “Typy” (pronounced: typey) on GitHub]]></summary></entry></feed>