This past week a missed security update contributed to a compromise at Matrix.org. As I have said before, for purposes of infrastructure design, it is prudent to consider CI/CD tools like Jenkins as “remote code execution as a service.” In the Continuous Delivery world, I think we have a serious problem with user education around securely running CI/CD tools; anything which can touch production represents a potential liability.

“Enterprise Software Sales” is not something I ever imagined spending as much time considering as I have over the past four years, but life is full of surprises isn’t it? At my previous gig we had changed our pricing model at least once during my time, and I learned quite a bit from the trade-off discussions which were had. Now sitting on the other side of the table, I get to enjoy a different perspective on the same underlying problem: how should enterprise software be priced? The question is important to answer, not just from a business perspective, but from a user perspective; the pricing model determines how your software will be adopted and used.

“But they weren’t doing true microservices” he droned on, while my train of thought came grinding a halt on the assertion. In my experience, many software developers apply all sorts of purity tests to the world around them, especially when it comes to “legacy” code. In most of my experiences, it has been delivered more subtly than this textbook example of the No true Scotsman fallacy. “Microservice” is already a silly term, one which many people defend by evoking the mythic status of “the unix philosophy.” Composition of components is definitely a valuable trait in a system, especially as an organization scales with new people and projects, but the microservice purity test fails in many cases.

The ability to shamelessly ask stupid questions has led me to numerous interesting projects and in some cases truly novel solutions. The subject of this blog post fits into the first part of that equation at least. I find the single static binaries produced by Rust and Golang to be quite compelling for system utilities, at the same time however I am fond of writing TypeScript. Why can’t I mix chocolate with my peanut butter?

Turn back now, this blog post is so niche that it’s statistically impossible for you to find this useful. Last night I was thinking about building a little app which needed to deal with an event stream, and started poking around the Azure Event Hubs documentation. I noticed that they apparently can now speak Kafka which means I can use my existing Kafka library tooling, nice! Since I was already working with Kafka and Rust for another little project, I took a quick detour and tried to see if I could publish to an Event Hub over Kafka, from Rust. As luck would have it, I can!

Building daemons and system-level utilities has always been something I have enjoyed. While I have professionally written C code, I have always found it a bit antiquated and unpleasant, like using a screwdriver while everybody around you is using power tools and machines. It certainly still has its place in the world, but there are more powerful options out there. I have experimented with Ada as a system level toolchain, while an all around compelling language it suffers from a severe lack of libraries and doesn’t have a strong community of tooling. Recently I started experimenting with Rust and despite it’s promise, it has been one of the most challenging languages to date for me to learn.

In the research Kohsuke, Tracy, and I did in the development of the Continuous Delivery Foundation, we learned a lot about how other free and open source foundations operate. I know more now than I had ever before about how the Eclipse Foundation, Apache Software Foundation, and numerous other LF-based foundations operate. One recurring theme which has come up has been the aversion to paying people to contribute code directly to the open source project. While not a universal pattern, looking to the FreeBSD Foundation which regularly issues grants for FreeBSD development, I am perplexed by this mindset in various foundations.

A number of people have asked me recently what I actually do for a living these days at Scribd. Due to the very public nature of my involvement with the Jenkins project and the Continuous Delivery Foundation, a few of my friends have seemingly forgotten that CI/CD is not actually my full time job! My career has largely been focused on two axis: building high-functioning engineering teams, and building backend API/service infrastructure.

Whether I’m sharing a locally developed service with a member of our globally distributed team, or I need to integrate some cloud-based service with local development, I frequently find the need to expose a local TCP service to the public internet. In the past I have tried to use tools such as localtunnel or smee.io, and in both cases I found them lacking; I simply want this TCP port open to the world! Yesterday afternoon I spent some time hacking on the first version of my own little solution: aci-tunnel.

Last week we announced the Continuous Delivery Foundation (CDF) at the 2019 Open Source Leadership Summit. Through a strange series of events I was fortunate to attend and participate in the “Continuous Delivery” keynote on the first day. Joining me on stage was Kohsuke (Jenkins), Kim and Christie (Tekton), Tracy (Jenkins X), and Andy (Spinnaker) to share a bit about the four initial projects joining the CDF.

The aviation community has been buzzing with speculation and commentary around the recently Boeing 737 MAX 8 plane crash in Ethiopia and the model’s subsequent grounding around the world. Watching this news report I was struck by the following quote from the “Deputy Assistant Secretary of State” regarding a similar crash in Indonesia:

Considering the percentage of my day which is spent typing on a keyboard, it should come as no surprise that I might have thoughts on what makes a “good” versus a “bad” keyboard. In fact, I think everybody who uses a tool with this level of frequency should have thoughts on what qualities make variations of the tool good or bad.

Today the Continuous Delivery Foundation officially launches, marking the completion of almost two years of work. Starting at the 2017 Jenkins World Contributor Summit where we, the Jenkins project discussed a “Jenkins Software Foundation”, to the 2018 Open Source Leadership Summit where the concept evolved into a continuous delivery focused organization, culminating in what we have today: a strong group of organizations and initial projects banding together for under the banner of the Continuous Delivery Foundation (CDF).

There are 90 days until the beginning of the week-long AIDS/LifeCycle, the 545 mile ride from San Francisco to Los Angeles to raise critically needed funds for HIV/AIDS-related services. For me, this means just under three more months to meet my fundraising goal. Not only that, this means my fellow riders and I have a shockingly small number of weekends to get our training in order!

The Jenkins project has long used Mirrorbrain, a great piece of software for running a high-traffic download site using redirect mirrors. We use it to transparently delegate traffic to a network of donated mirrosr, for downloading our Debian, Red Hat, and other packages of Jenkins and all of our plugins.

Linux has now been my primary desktop operating system for the better part of the last decade. Originally I had used openSUSE but found myself migrating to Debian for a number of reasons. I recently jumped back over to openSUSE, and have been impressed once again with the overall quality of the entire distribution.

I don’t think anybody can understate the value free and open source software has brought to the world at large, value which has largely been freely given with little expectation in return. The ubiquity of free and open source software seems to have fostered a sense of entitlement in the minds of some users. This presumption that free and open source software should do exactly what they expect it to do, and if not, that’s a problem that you the maintainer should address. I find this viewpoint to be not only incorrect, but abusive.

My exact words were “I could have likely filled half a book with thoughts and practices on good security for Jenkins” and were I to write that book, my colleague Daniel would certainly be a guest author of a few chapters, such as one on the limitations of credentials masking. In that linked post Daniel highlights some of the ways in which credentials can be misused in Jenkins. As I mentioned on Twitter, this is not a unique problem to Jenkins, any system which allows user-defined build processes and allows the use of credentials in those build processes will allow exposure of those credentials.

As the number of different ways to configure Jenkins approaches infinity, I have come to appreciate one pattern in particular as generally good. When I first started the draft of this blog post, three years ago (!), I wanted to share that by coupling Jenkins and Docker together I finally had the CI/CD server I had been waiting for. In the past few years, Docker has increasingly become the default environment for Jenkins, and rightfully so. In my previous post I mentioned some of the security pitfalls of using Docker in an untrusted CI/CD context, like that which we have in the Jenkins project. Regardless of trusted or untrusted workloads, I still think Docker is a key piece of our CI/CD story, and here I would like to outline why we need Docker in the first place.

Over the past few years, the topic of architecture and security for CI/CD environments has become among my favorite things to discuss with Jenkins users and administrators. While security is an important consideration to include in the design of any application architecture, with an automation server like Jenkins, security is crucial in a much more fundamental way than a traditional CRUD app. Walking that fine line between enabling arbitrary use-cases from developers and preserving the integrity of the system is a particularly acute problem for CI/CD servers like Jenkins.