The async/await keywords in modern Rust make building high-throughput daemons pretty straightforward, but as I learned that doesn’t necessarily mean “easy.” Last month on the Scribd tech blog wrote about a daemon named hotdog which we deployed into production: Ingesting production logs with Rust. In this post, I would like to write about some of the technical challenges I encountered getting the performance tuned for this async-std based Rust application.

Much of the information I read during the day, not counting e-mail, comes from my RSS reader: Newsboat. Whenever I see an interesting blog post on Twitter or elsewhere, I habitually subscribe the author’s RSS feed. I recently stumbled across an interesting RSS feed which wasn’t served over HTTP, leading me to wonder: how can I subscribe?

I discovered today that since version 8.1, Vim apparently supports spawning a terminal from within the Vim editor. This is a handy little feature that could make life easier for checking documentation, running tests, and so on.

I have been writing a lot of Rust lately and as a consequence I have had to get a lot better at writing unit tests. As if testing along weren’t tricky enough, almost everything I am writing takes advantage of async/await and is running on top of the async-std runtime.

Suddengly managing a remote engineering team may seem like a daunting situation, one which many people are suddently finding themselves in as tech companies institute sudden “work-from-home” policies in response to the Corona virus. If you find yourself in this situation don’t panic. Managing remotely is not significantly different than managing in-person, and your already existing good management and communication habits will greatly help. Nonetheless, I thought I might be able to help newly remote managers by hosting an open office hours, with the first experimental session yesterday in the afternoon PST.

If you are a sysadmin, Open Build Service is one of the tools you should add to your toolbox..today. “OBS”, hosted at build.opensuse.org is one of my favorite “killer apps” for openSUSE, yet for system administrators it has continued to be relatively unknown, but disproportionately valuable. At a high-level OBS is a tool for building and distributing packages, but on build.opensuse.org, there’s a social component which may someday save your bacon!

Build performance has always been important to me, but my pain tolerance has always varied widely depending on the project. The projects I have worked on which require the JVM, such as Jenkins or JRuby/Gradle, anything under 30 seconds seems amazing. For small Node and Ruby projects, anything over a few seconds feels atrocious. Since I’ve been hacking with Rust lately, I haven’t been able to figure out what constitutes “acceptable.” For my relatively small project, incremental compilation was very quick, but for some reason linking the project would talk almost 10 seconds. That seemed pretty unacceptable.

If the people I know tweet enough about something, eventually I’m bound to breakdown and just buy the thing. It happened with the Intel NUC, and now it’s happened with Yubikey. The Yubikey is a USB-based security device that can do a lot of things, but in my case I just need it to act as a security key for a number of websites such as GitHub, Google, and Twitter. Much to my dismay it did not work exactly as I expected right out of the box on my openSUSE-based laptop.

People will sometimes look at my screen covered in terminal windows overflowing with text: “how do read all that?” These moments remind me how inscrutable backend development can appear to the outsider. Today I would like to introduce a tool that I hope makes a some parts a little more easy to understand: Kafkakitty

This year I have been struggling to learn Rust, but I am now pleased to share that I’m finally understanding the language. Earlier I lamented the challenges of adopting Rust. Between semantically important apostrophes and angle-brackets a plenty, I was struggling to read and write basic Rust. I can easily read Ada, C, Python, JavaScript, Java, and Ruby. Something about the syntax of Rust remained difficult to process. The code looked jarring and dissonant, I could read snippets but translating entire functions or modules into a workable mental model was not feasible. Over the past month however, I believe I have made some progress up the learning curve. I can now write some Rust!

This week we launched the Scribd tech blog, on which I published today’s article: We’re building the largest library in history. I frequently have to remind myself that I have been here less than a year, and we have undergone incredible positive change, with more coming in 2020.

I have a love/hate relationship with containers. We have used containers for production services in the Jenkins project’s infrastructure for six or seven years, where they have been very useful. I run some desktop applications in containers. There are even a few Kubernetes clusters which show the tell-tale signs of my usage. Containers are great. Not a week goes by however when some oddity in containers, or the tools around them, throws a wrench into the gears and causes me great frustration. This week was one of those weeks: we suddenly had problems building our Docker containers in one of our Kubernetes environments.

A number of years ago I was building out a product with a small team, like most teams I’ve worked with, an irreverent sense of humor emerged. One of my colleagues quite enjoyed using the term “bro” ironically; he certainly was the type of person who wouldn’t come within earshot of any group of people who might use the term with any level of seriousness. As the product started to take shape, we found ourselves in need of fake users in our test system. I’m not sure who created this first user, but the user’s fullName was set to “Test Bro.” Shortly thereafter another user was added: “Broam Chomsky.”

TLS certificates have the largest “complexity/importance” scores imaginable. Everything about them is error prone and seemingly over-engineered from top to bottom, yet they are one of the most important pieces of security and authentication in our software architectures. From an engineering management standpoint, I am finding myself adopting the rule of: estimates for any project involving certificates should be multiplied tenfold. If the project involves the Java Virtual Machine (JVM) and the Java Key Store (JKS), multiply by another ten I suppose. For my own future convenience, in this blog post I would like to outline how to add a root certificate to a Java Key Store in Red Hat-derived environments.

Over the course of my professional career I have witnessed the transition from free and open source software being something useful engineers do, to a multi-billion dollar industry with companies jumping into the frenzy. During this time I have also gone from an open source user, to contributor, to a board member. Helping to steward a few small projects, but mainly focusing on the Jenkins project. Along the way I have interacted with businesses in each role, forming opinions of their businesses. Getting a sense of their cultural values by watching and listening as their employees interact with the project, or their executives make public statements about Jenkins or open source software in general. By night I am open source contributor, but by day I am now what enterprise sales people refer to as the “buyer.” One with opinions formed by years of interactions with these companies whose products we evaluate.

Running untrusted CI/CD workloads in Jenkins is perhaps my favorite security discussion. Throwing Docker into the mix makes things even interesting, and in some cases less secure. Today I implemented a pattern which I have discussed with colleagues but hadn’t yet had the opportunity to try: a multi-Kubernetes cluster for Jenkins. In short, running a Jenkins master in a cluster which acts as the control pane for it and many other services, while running all of its workloads in an entirely separate Kubernetes cluster. For those who know the joy of managing Kubernetes this may seem like madness, but it does offer a number of security benefits which I would like to outline.

My favorite part of the stack is the netherworld between the underlying infrastructure and the app. That fuzzy grey area where data goes from databases to object-relational mappers (ORMs), web servers to request libraries (e.g. Rack/WSGI), and so on. In many cases a technology roadmap where one considers infrastructure, but not the application, or vice-versa, is doomed from the start. At Scribd, I have been given permission to hire more people that love this layer of the stack, and I have taken to calling it “Ruby Infrastructure.” A phrase which is fairly unique, that I wanted to define in greater detail.

One of the harder parts about building new platform infrastructure at a company which has been around a while is figuring out exactly where to begin. At Scribd the company has built a good product and curated a large corpus of written content, but where next? As I alluded to in my previous post about the Platform Engineering organization, our “platform” components should help scale out, accelerate, or open up entirely new avenues of development. In this article, I want to describe one such project we have been working on and share some of the thought process behind its inception and prioritization: the Real-time Data Platform.

The team that I joined Scribd to build, Core Platform is now up and running with five incredibly talented people. I could not be more pleased with the very friendly and highly functional group of people we have been able to assemble. With that team’s projects underway, my focus has been shifting, zooming out to “Platform Engineering” as a comprehensive part of the engineering group. In this post, I want to expand on what Platform Engineering is planned to be and discuss some of the teams and their responsibilities.

Yesterday we rebuilt and re-deployed one of the Jenkins containers we use at work, and much to my chagrin the Jenkins environment no longer wanted to boot. We use Jenkins on top of Kubernetes, integrated with Hashicorp Vault, configured with the Configuration as Code plugin and the Job DSL plugin. While I am pleased with this stack of tools, it is not a “simple” set up. It had been three weeks since the last rebuild and redeploy, and the name of the game was: what of the dozen changes that have happened in one of these tools over the last three weeks was the culprit.