Howdy!

Welcome to my blog where I write about software development, cycling, and other random nonsense. This is not the only place I write, you can find more words I typed on the Buoyant Data blog, Scribd tech blog, and GitHub.

Vaccinated

Everything went fine. Social media is awash with anecdotes about getting vaccinated, side effects, and opinions about other people becoming vaccinated. I recently crossed the waiting threshold after my last dose and felt I should share my experiences as well.

Read more →

The kind of smart appliances I want

I want smart appliances, not the current commonly understood “smart” Internet-of-Things (IoT) appliances, but smart in actually useful ways. A couple years ago I had solar panels installed. The “smart” I want is pretty simple: I want devices that know about surplus energy. Devices which have lower power idle modes that can kick into more productive usage when solar power is bountiful. Generally IoT devices seem to be almost everything I don’t want. I don’t need devices that listen in on my conversations, track my data, and open up security holes in my home network. In this post I want to outline what I do want from “smart” appliances in my home.

Read more →

Ein Gespräch mit Dieter Nuhr

Deutsch ist nicht meine Muttersprache. Trotzdem höre ich seit Jahren deutschsprachige Podcasts. In den letzten paar Monaten habe ich Jung und Naiv gefunden und kann es euch nur empfehlen. Der Podcast-Feed ist eine Mischung aus Bundespressekonferenzen, der Hans Jessen Show, und Interviews mit verschiedenen Leute in der oder aus der Politik. Viele von diesen Interviews sind wahnsinnig lang und ich habe selten ein paar Stunden Zeit, wo ich die Kopfhörer rein stecken kann um ein einziges Gespräch am Stück zu hören. Deswegen habe ich mir in den vergangenen Wochen in kleinen Teilen dieses Interview mit Dieter Nuhr angehört.

Read more →

Preparing for Delta Hack: 2021

This week is Data and AI Summit at which I am speaking, but I’m already looking past it at Delta Hack: 2021. My interest in using Delta Lake has been steadily increasing since we started delta-rs and I’m finding more and more use-cases for our recently introduced writer support. Delta Hack (#deltahack) will be an opportunity to really dive into some more complex problems that cannot be solved in an hour or two after work.

Read more →

Converting XML to JSON in Rust

I generally default to using JSON for data interchange but there are still a myriad of formats of XML out there, for which I have created the xmltojson crate. I originally wrote this one night to help me get an XML dataset into JSON so that I could use PostgreSQL’s JSONB column type, but I only recently published it to crates.io since it may be useful for others.

Read more →

Dynamically forwarding SSH ports

Working over SSH on a big beefy remote machine is a great way to extend the life of any laptop, but presents challenges when developing network-based services. Fortunately OpenSSH has a “port forwarding” feature which has been around for a number of years. Port forwarding allows the user to tunnel a port from the remote machine back to their local machine, in essence allowing you to access a remote service bound to port 8000 on your own localhost:8000. When I first learned about this, I would fiddle around with my ssh invocations or hardcode a list of potential ports forwarded in my ~/.ssh/config. If I was working on a new service that needed a port not yet forwarded, I would disconnect, add it to the list of ports in my config file, and then reconnect. That was until my pal nibz (nibalizer) showed me how to dynamically add port forwards to an already existing session.

Read more →

Generating pre-signed S3 URLs in Rust

Creating Pre-signed S3 URLs in Rust took me a little more brainpower than I had anticipated, so I thought I would share how to generate them using Rusoto. Pre-signed URLs allow the creation of purpose built URLs for fetching or uploading objects to S3, and can be especially useful when granting access to S3 objects to mobile or web clients. In my use-case, I wanted the clients of my web service to be able to access some specific objects from a bucket.

Read more →

Why build a native interface to Delta Lake

Investing in the development of delta-rs is one of the longer shots I have taken recently and with my upcoming Data and AI Summit talk on the subject, I wanted to share some back story on the project. As I have mentioned before Delta Lake is a key component of Scribd’s data platform. We selected Delta Lake for many reasons, including that it is a open and vendor neutral project. The power of Delta Lake has opened up countless opportunities for data and for the past year I have seen the potential for many more.

Read more →

Recovering from disasters with Delta Lake

Entering into the data platform space with a lot of experience in more traditional production operations is a lot of fun, especially when you ask questions like “what if X goes horribly wrong?” My favorite scenario to consider is: “how much damage could one accidentally cause with our existing policies and controls?” At Scribd we have made Delta Lake a cornerstone of our data platform, and as such I’ve spent a lot of time thinking about what could go wrong and how we would defend against it.

Read more →

Understanding big data partitioning

Data partitioning is one of the principles to utilize when developing large data sets, but do you know what that actually means for the storage format? I didn’t! Many “big data” storage systems such as HDFS, S3, and Azure Data Lake Storage all are effectively a file system. This past year or so, I’ve become much more familiar with Delta Lake and kind of just assumed that data partitioning was something being done at the transaction log level. Turns out I guessed wrong.

Read more →