Getting over the hump required more effort than I had hoped, but after some small practice exercises I had the confidence and skills to move on to less trivial Rust. I am not one for solving Project Euler problems but instead I focused on porting little utilities which I regularly use in my local environment. The main educational benefit of these utilities is that none of them required network I/O, which is what always really interests me about writing software. Instead these utilities have very fixed inputs and outputs. A small well-understood problem domain made it much easier for me to clear my head and focus on the reading and writing of Rust.

I also purchased a paper copy of the Rust Programming Language book. I’m not one for reading programming books cover-to-cover, but having a paper copy still turned out to be useful. Taking the paper copy to another location, away from the computer, allowed me to focus on a few key chapters which addressed key areas of my confusion. The book is written quite well, plenty of strong narrative explaining concepts behind Rust rather than solely focusing on code snippets and the explanation that surrounds them. Among other things, the narrative helped me wrap my brain around the zen and art of ownership, a key concept to Rust if you’re to build anything consequential with it.

Finally, I have started to read more Rust, which also helped me up the learning curve. The convention of an examples/ directory in Crates has proven itself invaluable. Example snippets for crates which do interesting things, or interface with systems that I am also interested in working with, gave me a better feel of the syntax, structure, and overall aesthetic of Rust.

I still struggle with the myriad of built-in modules in the standard library, but I know that to be unavoidable from my time in the Java ecosystem. As one writes more and more, slowly but surely the standard library seeps into the muscle memory as if through osmosis. It just takes time.

Perhaps the learning curve would not have felt so steep if I had been able to dedicate a week or two to the exercise of learning a new and structurally different systems programming language. Once upon a time I could have dedicated that much effort to such a hobby but those days are long gone now. Instead I find myself picking up an hour here, thirty minutes there, and trying to load mental context fast enough to make some progress with learning a new skill. That approach might work for simple things, but Rust definitely was not that to me.

If you are an experienced developer learning any new language of sufficient newness, I would encourage you to think about how you learned to program in the first place. Simple, small, almost embarrassingly basic exercises. The complex systems you are capable of modeling will have to wait, you may have to go back to the beginning. Back to the third, fourth, and fifth exercises after your “Hello World.” It is painful, but this time around it will go much faster than the first.

Unlike my previous post on dynamic task creation in Ada, this post won’t have much in the way of sample code and walk-through. Mostly because I didn’t want to spend the entire post explaining epoll(7) concepts.

If you’re currently unfamiliar with epoll, I found this post from banu.com interesting and useful. In general, I think that it is very useful to understand how both epoll(7) and its BSD-friendly pal kqueue(2) work, and how the two have influenced evented I/O systems that have become popular in the past few years.

For my experiment, the primary challenge with using epoll(7) is wrapping the C code with Ada, which on its face this isn’t terribly difficult. That said, mapping C arrays and other stupid pointer tricks that C libraries are prone to using, can make writing an Ada binding much more difficult. I don’t intend to dive too deep into how one can create such a binding since I think Felix Krause’s post on writing thin/thick C bindings in Ada is a pretty good introduction on that subject. Personally I prefer “thicker” bindings for use in Ada, just as I expect Ruby libraries to abstract away the more fickle aspects of the underlying C API.

What I did want to share in this post was the actual working results of my tinkering, which can be found here, on GitHub.

The basic flow of the program (found in main.adb) is similar to the example found in the epoll(7) man page, in that it sets up the epollfd, binds the listening to it and then waits for Epoll.Wait to return with an array of socket descriptors which have activity on them.

My next step is to combine this basic epoll binding with my previous task related code and create a TCP server which uses a task pool to perform “work” on the incoming sockets given to it by an epoll-listener task.

(If you’re familiar with Ada, you can skip this next section)

Note: You can find all this code, and more in my ada-playground repository on GitHub

Similar to C, Ada supports stack allocated variables as well as heap allocated variabels, it also defaults to stack allocation. For example:

    procedure Main is
        -- A stack allocated `Integer` object
        Enough_Memory : constant Integer := 655360;
    begin
        null;
    end Main;

If you wanted to allocate that Integer onto the heap, then you would use the new keyword:

    procedure Main is
        -- A heap allocated `Integer` pointer
        Enough_Memory : access Integer := new Integer'(655360);
    begin
        null;
    end Main;

I won’t dive too much into the minutia of what is going on here, if you’re not familiar with Ada already you can learn more about access types on the Ada Programming Wikibook. Basically we’re heap allocating a new Integer and using an access type (aka: typed pointer) to keep track of it. Keen readers will notice we didn’t do anything with that Integer access type, and we’re technically leaking the memory. To solve this we use the generic unit Ada.Unchecked_Deallocation, which gives you a facility for properly freeing memory (more details here).

Tasking Trickiness

Concurrency is part of the language in Ada, and is handled through tasking. A basic example of might be:

    with Ada.Text_IO;
    procedure Main is
        task Counter;
        task body Counter is
        begin
            for Count in 1 .. 10 loop
                Ada.Text_IO.Put_Line (Count'Img);
            end loop;
        end Counter;
    begin
        null;
    end Main;

The way tasks in Ada work means that the Counter task will be created, started and then the execution of the Main program will block until the Counter task completes (important detail).

The trickiness starts to arrive when you talk about dynamically allocating task objects, and combine that with something like an infinite loop, such as one might find in a server program, e.g.:

    procedure Main is
    begin
        -- Socket set up omitted
        loop
            declare
                Client_Socket : Socket_Type;
                Request_Handler : Handler_Ptr := new Handler;
            begin
                -- Block until we receive a new inbound connection
                Accept_Socket (Server_Socket, Client_Socket, Server_Addr);
                -- Dereference the Handler_Ptr and call `Process` on the Handler
                -- task
                Request_Handler.all.Process (Client_Socket);
            end;
        end loop;
    end Main;

(The code above is an abbreviated version of echomultitask_main.adb which can be found here).

The issue with this code is that we’re allocating a new Handler task for every in-bound connection, and we have no means of ever cleaning them up properly. If we were to create an Array of Handler_Ptr, we still would have to find some mechanism (which exists) to check the status of each Handler to determine if we should clean it up. Problem being, we’d have to loop through all the active tasks, checking for a “terminated” status, in order to deallocate them. It’d be much better if a task could tell us when it’s finished, rather than us polling every one.

Fortunately in Ada 2005, a mechanism was added to make it easier to add “clean-up” to tasks: Ada.Task_Termination. The package allows you to set up a termination handler for the a specific task, which the runtime will call when that task terminates. Unfortunately however, the handler procedure that can be invoked when the task terminates will not be passed a pointer to the task itself, but rather the Task_Id (Ada.Task_Identification.Task_Id).

So close to being able to properly deallocate these dynamic tasks, but we need one more component, a protected object with a hash map inside of it:

    package Server.Handlers is
        protected Coordinator is
            procedure Track (Ptr : in Handler_Ptr);
        private
            Active_Tasks : Handler_Containers.Map;
        end Coordinator;
    end Server.Handlers;

Then back in main.adb:

    Accept_Socket (Server_Socket, Client_Socket, Server_Addr);
    Request_Handler.all.Process (Client_Socket);
    -- Make sure the we keep track of the Request_Handler in order to properly
    -- deallocate it later
    Server.Handlers.Coordinator.Track (Request_Handler);

(The code above is an abbreviated version of echomultitask-worker.ads which can be found here)

The singleton protected object Coordinator not only will give us protected (aka thread safe) access to the Active_Tasks map, but also gives us a protected object to hang our Ada.Task_Termination.Termination_Handler protected procedure off of:

    protected body Server.Handlers is
        protected body Coordinator is
            procedure Last_Wish (C : Ada.Task_Termination.Cause_Of_Termination;
                                T : Ada.Task_Identification.Task_Id;
                                X : Ada.Exceptions.Exception_Occurrence) is
            begin
                -- Deallocate our task identified by `T`
                -- and make sure we remove it from `Active_Tasks`
            end Last_Wish;
            procedure Track (Ptr : in Handler_Ptr) is
                -- Dereference our `Handler` task, and fish out its Task_Id
                Handler_Id : Ada.Task_Idenfitication.Task_Id := Ptr.all'Identity
            begin
                -- Add `Handler_Id` to our `Active_Tasks` map
                Active_Tasks.Insert (Handler_Id, Ptr);
                -- Set up the Last_Wish procedure to be executed after our task has
                -- terminated
                Ada.Task_Termination.Set_Specific_Handler (Handler_Id, Last_Wish'Access);
            end Track;
        end Coordinator;
    end Server.Handlers;

(The code above is an abbreviated version of echomultitask-worker.adb which can be found here)

This approach will allow us to safely create new dynamic tasks to handle the incoming requests, but will also make sure that the tasks are cleanly deallocated when they terminate.

If you’re interested in concurrency in Ada, I highly recommend purchasing Concurrent and Real-Time Programming in Ada by Alan Burns and Andy Wellings, it’s been tremendously helpful for my own concurrency exploration in Ada.

Inspired, I figured I would try out the same set up on my own. The first major hurdle to overcome was actually streaming my desktop to Justin.tv. After a full evening of searching, and tinkering with ffmpeg, I managed to get this script working well enough to send audio and video across the internets.

With the “infrastructure” figured out, I started hosting a couple of sessions while I worked on projects that I’m passionate about. I found the live coding concept immediately useful when I managed to get live help from an engineer at Puppet Labs while I dug through the Puppet::AST parsing code.

Taking the experiment further, I even tried helping some folks on IRC with “live support sessions” which were so fruitful that I posted a couple videos (1, 2) from them. In the office, I can say “hey, come over here and look at this” when others need help, and the live coding stream allowed me to extend that concept to practically anybody I wished to help.

I haven’t helped anybody in a few months now, I use my typically viewer-less sessions to help me focus on a single project. Whenever I turn the stream on, I’m pair-programming with a companion who may or may not be there (considering the #codingwithrtyler channel on Freenode is generally empty, I’m guessing there’s nobody watching), but I focus just as if I had another engineer sitting next to me.

It’s almost like a weird hybrid of pair programming and rubber duck debugging now that I think about it.

The projects I work are not nearly as visually pleasing as Minecraft, and I don’t have the geek fame that Notch has but I’m perfectly content to stream my desktop to zero viewers as long as it proves useful for evening and weekend hack-sessions.

Here’s a boring demonstration:

I’ll save my thoughts on how the experiment is going for a later blog post, in this post I just wanted to share the how.

First I use this ~/bin/screenstream script:

#!/bin/sh -xe

INFO=$(xwininfo -frame)
API_KEY="YOUR_API_KEY_GOES_HERE"

WIN_GEO=$(echo $INFO | grep -oEe 'geometry [0-9]+x[0-9]+' | grep -oEe '[0-9]+x[0-9]+')
WIN_XY=$(echo $INFO | grep -oEe 'Corners:\s+\+[0-9]+\+[0-9]+' | grep -oEe '[0-9]+\+[0-9]+' | sed -e 's/\+/,/' )
FPS="15"

INRES='1680x1010'
OUTRES='1280x720'

ffmpeg -f x11grab -s "$INRES" -r "$FPS" -i :0.0+$WIN_XY \
    -f alsa -ac 2 -i default -vcodec libx264  -s "$OUTRES"  \
    -acodec libmp3lame -ab 128k -ar 44100 -threads 0 \
    -f flv "rtmp://live.justin.tv/app/$API_KEY"

There’s a couple important things to mention about this:

• The xwininfo invocation just gives me a cross-hairs cursor so I can select which window area I want to record. This doesn’t technically restrict ffmpeg(1) to just that window, rather it just grabs the offset from the window and uses those.
• The segment: -f alsa -ac 2 -i default pertains to the audio input. According to arecord -L, the pulse audio sink I should use is called “default”, on some machines it might be called “pulse”, your mileage may vary.
• I use the pavucontrol (GUI) tool to direct my audio out to the pulseaudio input, this allows me to share what I’m listening to with whoever is watching the stream.
• You’ll notice the $INRES and $OUTRES parameters, I am techically live-resizing the video on the fly, which takes up a lot of CPU power, you may or may not want to do this depending on your machine speed, size of your screen and the amount of desktop space you want to stream.

That’s about all there is too it, I wish I could either take credit for the script or at least attribute it, but I can do neither because I found it on a forum some where and then quickly lost the link. Whoops.

“Why do we need tools like Puppet or Chef when we have XML?”

This question was relayed to me second-hand, so I apologize if I’m butchering it. The question itself is so mind-boggling that I can only assume one of three things about the asker:

• He/she is a rather talented troll
• He/she has some crucial misunderstandings about configuration management, in which case I’m hoping somebody politely explained things.
• He/she understands CM but has a serious fetish for XML.

Regardless of their intention, I have them to thank for inspiring what might be the innovation breakthrough of 2012:

Puppet/XML

I’ve long found Puppet’s DSL syntax to be too sparse, easy to type and semantically boring, take for example this block:

class users {
    group {
        "tyler" :
            ensure => present;
    }
    user {
        "tyler" :
            require => Group["tyler"],
            ensure  => present;
    }
}

How terrible! Let’s turn that into Puppet/XML!

<?xml version="1.0"?>
<puppet>
    <classes>
        <class name="users">
            <resources>
                <group name="tyler">
                    <ensure>present</ensure>
                </group>
                <user name="tyler">
                    <requires>
                        <require>Group["tyler"]</require>
                    </requires>
                    <ensure>present</ensure>
                </user>
            </resources>
        </class>
    </classes>
</puppet>

Not only is that easier to understand, it’s more enterprise ready!

Currently Puppet/XML is in pre-alpha and in this repository on GitHub. Thus far it’s only been tested with puppet apply (i.e. standalone mode) under Vagrant, an example run is below:

[default] Running Puppet with /tmp/vagrant-puppet/manifests/base.ppx...
stdin: is not a tty
Thanks for using Puppet/XML - A brand new synergy

info: Retrieving plugin
info: Applying configuration version '1328719113'
notice: /Stage[main]/Users/User[tyler]/ensure: created
notice: /Stage[main]//Node[default]/Group[puppet]/ensure: created
info: Creating state file /var/lib/puppet/state/state.yaml
notice: Finished catalog run in 0.62 seconds

Stay tuned for new innovations and more middleware that your enterprise can leverage to help meet your business objectives.

Disclaimer: Unfortunately the code currently relies on transcribing the Puppet/XML to Puppet’s “native” DSL, this means more complex things like conditionals and function calls are not yet supported. After spending about an hour and a half trying to generate the appropriate Puppet::AST structures, I gave up and went the route which guaranteed lulz sooner.

tyler:  man, I am getting that "I should really learn smalltalk"
        feeling again
tyler:  everytime I get close I find myself downloading a squeak
        VM and then going "LOLWTF IS THIS SHIT"

I said this before I left the office and somewhere in the back of my mind I kept thinking “why can’t I crack Smalltalk?” This continued through most of the evening until 9 pm (21:00) rolled around and I decided that I wouldn’t let that commie Alan Kay defeat me.

After finding this fantastic series of screencasts I set about learning some Smalltalk. Not enough to build anything major, but enough to really understand Smalltalk’s core concepts.

In exploring the world of Squeak, Pharo and Amber I believe I have experienced the full spectrum of emotions that my circuits provide, all the way from unfettered rage to mild bemusement. Despite all of this, around 2 a.m. I found myself lying awake in bed wanting to marvelling at the design of the language and wanting to learn more.

A couple of things still bug the hell out of me:

• The VM concept while novel is a completely jarring experience for anybody who has done any programming in any other environment ever.
• I hate mice. I also have no trackpad on my laptop which means if i’m not at my desk using a mouse-heavy program becomes painful (quite literally). Both Squeak and Pharo have this sick mouse fetish that makes me want to （╯°□°）╯︵┻━┻
• GNU/Smalltalk feels closer to development environments that I’m familiar with, but VisualGST is incredibly buggy and I think part of what makes Smalltalk special is the integrated “live environment.”

The things that I do really like however:

• The integrated “live environment” is something out of programmer fairy tales. Inspect/debug/edit anything and everything? Yes please!
• Everything is an object which receives messages, the syntax for conditionals helps drive it home. A boolean object can receive an ifTrue or an ifFalse message which takes a block argument. This means there is no real for if statements, it’s all implemented with the basic Smalltalk primitives: 1 < 3 ifTrue: [Transcript show: ‘Truthy!’] ifFalse: [Transcript show: ‘Untruthy!’] How about a for loop? Surely the language must have a for loop! No way José! Instead an Array object just takes a do message with a block, fancy! #(‘tom’ ‘dick’ ‘harry’) do: [ :each | Transcript show: (‘Hello ‘, each); cr]
• No magic, that I’ve seen thus far. One of my biggest criticisms of Ruby comes out of the Rails tradition of magic methods by using method_missing to magically generate methods on-demand, leaving a system that’s abominably difficult to inspect and debug. Thus far I’ve not found any slight-of-hand in Smalltalk, which is a good thing.

I can really see where Objective-C pulled from Smalltalk having now experienced some Smalltalk myself. That said, it’s rather unfortunate that C and some additional custom bits have creeped into Objective-C making it less Smalltalky than it once was.

I am definitely going to continue to tinker with Smalltalk, likely with Pharo as my VM of choice. My only concern is that I will grow to resent the other languages I use regularly which feel dumber than a language invented almost 40 years ago.

Update: This is the original post that pushed me over the edge on tinkering with Smalltalk.

The only downside I’ve found, thus far for Cucumber comes down to the use of the term “cukes” to describe Cucumber-based tests.

The word “cuke” or “cukes” simply has an unpleasant, wretched feeling to it. It sounds like something unpleasant to get on your shoes. I really do not like this word.

As such, I will henceforth refer to Cucumber-based tests as “gherkins” instead.

I look forward to writing plenty of posts in the future about creating high quality gherkins for browser testing and cross-system testing, stay tuned!

(I sure have been complaining about names a lot lately haven’t I?)

I’ve slightly changed my opinions since then, mostly because I can no longer afford to avoid JOINs. Sometimes harsh reality is the best cure for young idealism.

Anywho, what was my point again about joins?

Ah yes, joins in ActiveRecord. While ActiveRecord is a very solid ORM gem for Ruby, it is not without it’s quirks. Most of which come down to “magic” behavior or defaults you’ve assumed incorrectly. As much as I would like to pontificate on the odd method_missing magic ActiveRecord developers seem to love, this post is more about the latter.

In general, JOINs involve two tables, JOINing together the tables typically for a nested query of some form or fashion. For an example, let’s use this make believe query:

    SELECT `guests`.* FROM `guests` JOIN `plates` \
        ON plates.guest_id = guests.id WHERE \
        (guests.favorite = 'sushi' OR plates.content = 'sushi');

There are five basic JOINs in SQL land, and in this context they mean:

• LEFT JOIN: Find all the rows in guests, even if they don’t have a corresponding plates row, then apply sub-query.
• RIGHT JOIN: Find all rows in plates, even if they don’t have a corresponding guests row, then apply sub-query.
• INNER JOIN: Find rows only if guests has a corresponding plates row.
• STRAIGHT JOIN: Find your way to the DBA’s desk and receive your punishment.
• NATURAL JOIN: I’ll be honest, I don’t fully understand what a natural join does.

Back in ActiveRecord land, let’s imagine I want to create a named_scope, which in Rails 3 has been deprecated in favor of just scope, I’d write code something like this:

    class Guest < ActiveRecord::Base
      has_one :plate
      # ...
      named_scope :with_food, lambda { |food| {
                    :conditions => ["guests.favorite = ? OR plates.content = ?", food, food],
                    :joins => :plate
                }}
    end

Underneath the hood, ActiveRecord generates a query almost exactly like the one above with one subtle difference. “JOIN” is instead an “INNER JOIN” which means my query will not return a Guest object for any guests who do not already have a plate object.

The “solution” is to use a LEFT JOIN, which is unfortunately rather gnarly. There may be a better way to perform alternate JOINs in ActiveRecord, but I don’t yet of one:

    class Guest < ActiveRecord::Base
      has_one :plate
      # ...
      named_scope :with_food, lambda { |food| {
                    :conditions => ["guests.favorite = ? OR plates.content = ?", food, food],
                    :joins => "LEFT JOIN `plates` ON plates.guest_id = guests.id"
                }}
    end

Basically if you want to use anything other than a simple INNER JOIN, you’ve got to enter it in yourself. At a certain point ActiveRecord throws up its hands and says “Look, buddy, I have no idea what you’re trying to do here. Enter your own damn SQL.”

For better or worse, Lookout has standardized on Ruby 1.8 in which I’ve discovered one absolutely infuriating “quirk” with how IO#popen handles pipe redirection and spawning processes. A “quirk” that has managed to screw me in more than two projects thus far. While the issue affects both Ruby 1.8 and Ruby 1.9, there is no work-around in Ruby 1.8, which is especially frustrating.

When you execute IO#popen, you receive an IO object which implements some methods which allow you to read stdout from the child process. Pretty reasonable, sort of I guess, except that the new child process inherits the parent’s stderr pipe. From the process table point of view, you will see exactly what you expect:

3046 pts/7    Ss     0:00 ruby demo.rb
9521 pts/7    R+     0:00  \_ run_daemon -f some.conf --verbose

Naturally you might want to redirect all the pipes in order to segment the output of the spawned child process from the parent process. If you want to separate them, i.e. a child_stdout and a child_stderr pipe, then the Open3#popen3 has you covered.

What if you want to combine the two pipes (!) for reading inside of your parent process? If for example, you’re wrapping a process and running particular code depending on the collective output of the subprocess.

IO#popen can do this with the magical “2>&1” token in your command, e.g.:

def spawn_and_watch
  @child = IO.popen("run_daemon -f some.conf --verbose 2>&1")

  Thread.new do
    while @child.gets |line| do
      # do magic
    end
  end
end

So that’s pretty disgusting just from the code standpoint. What’s even more disgusting about this is what happens underneath the hood when you add the “2>&1” token into the command:

3046 pts/7    Ss     0:00 ruby demo.rb
9521 pts/7    R+     0:00  \_ sh -c run_daemon -f some.conf --verbose 2>&1
9522 pts/7    R+     0:00      \_ run_daemon -f some.conf --verbose

Subtle isn’t it?

This has wide-reaching consequences for the caller of IO#popen, the returned IO object inside of my make believe spawn_and_watch method has the pid of the sh executable. On top of that, to the best of my knowledge there is no way to actually get a handle on the run_daemon process ID that is now a grand-child process of ruby demo.rb. If I do something as foolish as call: Process.kill("TERM", @child.pid) then I will orphan my actual process and not get anything close to my desired behavior.

Basically, using “2>&1” with IO#popen is only good if you want to orphan processes, and prevent yourself from ever being able to send any signals to the process you want. This handling of commands passed to IO#popen with the “2>&1” token is consistent(ly bad) in both Ruby 1.8 and Ruby 1.9.

Fortunately for those using Ruby 1.9, you can work-around this behavior by changing the above method to look like this:

def spawn_and_watch
  @child = IO.popen(["run_daemon", "-f", "some.conf", "--verbose",
                                :err => [:child, :out]])
  # ...
end

It is because of this behavior alone that I have switched some personal projects to default to Ruby 1.9.2 and above, which is nuts.