I want smart appliances, not the current commonly understood “smart” Internet-of-Things (IoT) appliances, but smart in actually useful ways. A couple years ago I had solar panels installed. The “smart” I want is pretty simple: I want devices that know about surplus energy. Devices which have lower power idle modes that can kick into more productive usage when solar power is bountiful. Generally IoT devices seem to be almost everything I don’t want. I don’t need devices that listen in on my conversations, track my data, and open up security holes in my home network. In this post I want to outline what I do want from “smart” appliances in my home.

Deutsch ist nicht meine Muttersprache. Trotzdem höre ich seit Jahren deutschsprachige Podcasts. In den letzten paar Monaten habe ich Jung und Naiv gefunden und kann es euch nur empfehlen. Der Podcast-Feed ist eine Mischung aus Bundespressekonferenzen, der Hans Jessen Show, und Interviews mit verschiedenen Leute in der oder aus der Politik. Viele von diesen Interviews sind wahnsinnig lang und ich habe selten ein paar Stunden Zeit, wo ich die Kopfhörer rein stecken kann um ein einziges Gespräch am Stück zu hören. Deswegen habe ich mir in den vergangenen Wochen in kleinen Teilen dieses Interview mit Dieter Nuhr angehört.

This week is Data and AI Summit at which I am speaking, but I’m already looking past it at Delta Hack: 2021. My interest in using Delta Lake has been steadily increasing since we started delta-rs and I’m finding more and more use-cases for our recently introduced writer support. Delta Hack (#deltahack) will be an opportunity to really dive into some more complex problems that cannot be solved in an hour or two after work.

I generally default to using JSON for data interchange but there are still a myriad of formats of XML out there, for which I have created the xmltojson crate. I originally wrote this one night to help me get an XML dataset into JSON so that I could use PostgreSQL’s JSONB column type, but I only recently published it to crates.io since it may be useful for others.

Working over SSH on a big beefy remote machine is a great way to extend the life of any laptop, but presents challenges when developing network-based services. Fortunately OpenSSH has a “port forwarding” feature which has been around for a number of years. Port forwarding allows the user to tunnel a port from the remote machine back to their local machine, in essence allowing you to access a remote service bound to port 8000 on your own localhost:8000. When I first learned about this, I would fiddle around with my ssh invocations or hardcode a list of potential ports forwarded in my ~/.ssh/config. If I was working on a new service that needed a port not yet forwarded, I would disconnect, add it to the list of ports in my config file, and then reconnect. That was until my pal nibz (nibalizer) showed me how to dynamically add port forwards to an already existing session.

Creating Pre-signed S3 URLs in Rust took me a little more brainpower than I had anticipated, so I thought I would share how to generate them using Rusoto. Pre-signed URLs allow the creation of purpose built URLs for fetching or uploading objects to S3, and can be especially useful when granting access to S3 objects to mobile or web clients. In my use-case, I wanted the clients of my web service to be able to access some specific objects from a bucket.

Investing in the development of delta-rs is one of the longer shots I have taken recently and with my upcoming Data and AI Summit talk on the subject, I wanted to share some back story on the project. As I have mentioned before Delta Lake is a key component of Scribd’s data platform. We selected Delta Lake for many reasons, including that it is a open and vendor neutral project. The power of Delta Lake has opened up countless opportunities for data and for the past year I have seen the potential for many more.

Entering into the data platform space with a lot of experience in more traditional production operations is a lot of fun, especially when you ask questions like “what if X goes horribly wrong?” My favorite scenario to consider is: “how much damage could one accidentally cause with our existing policies and controls?” At Scribd we have made Delta Lake a cornerstone of our data platform, and as such I’ve spent a lot of time thinking about what could go wrong and how we would defend against it.

Data partitioning is one of the principles to utilize when developing large data sets, but do you know what that actually means for the storage format? I didn’t! Many “big data” storage systems such as HDFS, S3, and Azure Data Lake Storage all are effectively a file system. This past year or so, I’ve become much more familiar with Delta Lake and kind of just assumed that data partitioning was something being done at the transaction log level. Turns out I guessed wrong.

This weekend I finally got around to building a little Rust “full text search engine” based on the educational post written by my Scribd colleague Bart: titled Building a full-text search engine in 150 lines of Python code. Bart did a great job writing an accessible post which introduced some common search concepts using Python, my objective wasn’t necessarily to write something faster or better but to use the exercise as Rust practice. My day job is no longer writing code so the opportunity for a problem with fixed scope which would work out my Rust muscles was too good to pass up. In this post I want to share some things which I’ve learned in the process of duplicating Bart’s work.