just use postgre0

Automated anomaly detection has become invaluable in the observability tools I rely on for production systems. More advanced vendors like Datadog have introduced “AI” functionality, which is really just conventional machine-learning models for proactively identifying suspicious behavior.

One of the first advanced concepts taught to people learning to code is abstraction, practicing taking logic and hiding it. Object-oriented programming articles and books frequently will encourage the reader to take logic and complexity and build simpler APIs around it. Abstracting and encapsulating that logic helps us offload cognitive burden and allows us to focus on other aspects of the program. Abstraction is useful. Abstraction is good.

One of the reasons I work where I work is because of the fascinating data-at-scale problems that they have. This has led me deep into the world of Delta Lake and AWS S3. Not one to take anything too seriously, I have been cooking up absolutely bonkers solutions to some of these billions-scale challenges I am tasked with solving.

The wisdom that experience can bring means I now look at statements like “organizing the world’s information” through a very different lens. Google’s mission statement was something I did not view as coercive, because I didn’t understand the power exerted by those who collect and organize data.

I was lucky enough to record a podcast episode with Daniel Beach of Data Engineering Central. Daniel and I have known each other for a couple years sharing notes and ideas on the state of the ecosystem, where it falls down, and where things are getting interesting.

Cycling in northern California is something special. Over the past few decades Sonoma county has produced numerous professional mountain, road, and cyclocross athletes. Cycling is an institution, everybody is riding the roads or the mountains around here. It is not uncommon to see beat-up work trucks, Rivians, sensible sedans, and everything in between parked near some of our mountain biking trails.

Working in the data storage and services it can seem like everything revolves around capacity and throughput. We don’t think of throughput until it is lacking. A traffic jam, a flipped breaker, or an overflowing drain. There are architectural changes we make to improve throughput and there are tactical fixes. This post is about the tactical fixes.

The rate of change for data storage systems has accelerated to a frenzied pace and most storage architectures I have seen simply cannot keep up. Much of my time is spent thinking about large-scale tabular data stored in Delta Lake which is one of the “lakehouse” storage systems along with Apache Iceberg and others. These storage architectures were developed 5-10 years ago to solve problems faced moving from data warehouse architectures to massive scale structured data needs faced by many organizations. The storage changes we need today must support “multimodal data” which is a dramatic departure in many ways from the traditional query and usage patterns our existing infrastructure supports.

The Delta Kernel is one of the most technically challenging and ambitious open source projects I have worked on. Kernel is fundamentally about unifying all of our needs and wants from a Delta Lake implementation into a single cohesive yet-pluggable API surface. Towards the end of 2025 TD asked me to jot down some of the issues which have been frustrating me and/or slowing down the adoption of kernel in projects like delta-rs. At the outset of the project we all discussed concerns about what could actually be possible as we set out into uncharted territory. In many ways we have succeeded, in others we have failed.

On a day-to-day basis I build a lot of Rust code. To make my life easier I use sccache which I have written about previously. Periodically the sccache daemon would exit and then no longer authenticate against my local network’s not-S3 service.

All open source code has conditions attached. The majority of code which I have written in my lifetime has been open source and therefore is usually available for you to build from, distribute, or derive new works. There are some stipulations however and in this post I would like to help you understand how you can take code I have written.

In theory many developers understand concurrency and parallelism, in practice I think almost none of us do. At least not all the time. Building a mental model of highly parallel interdependent software is incredibly time-consuming, difficult, and error-prone. I have recently been doing a lot of performance analysis with both delta-rs and delta-kernel-rs. In the process I have had to check some of my own assumptions of how things should work compared to how they do work.

I would guess most developers think of URLs as a string with a https:// at the beginning. In many cases there are assumptions that are made about these URL-shaped strings which may be confusing, misleading, or flat out incorrect. The url crate is compliant to the RFCs about URLs, but while being technically correct is the best kind of correct, that doesn’t mean it still isn’t confusing.

On a small crate I maintain a friendly stranger made a suggestion to improve performance, by making logging optional.

After five years in production kafka-delta-ingest at Scribd has been shut off and removed from our infrastructure. kafka-delta-ingest was the motivation behind my team creating delta-rs, the most successful open source project I have started to date. With kafka-delta-ingest we achieved our original stated goals and reduced streaming data ingestion costs by 95%. In the time since however, we have further reduced that cost with even more efficient infrastructure.

Did you know that AWS S3 is almost 20 years old? The “cloud” as a concept is fairly recent but in the time-distortion that has occurred since the rise of the internet, I think many of us have lost track of how old some of these public cloud providers are, and as a side-effect, how old their technology offerings can become. Periodically you need to clean out the attic, and this week AWS did just that with their “AWS Service Availability Updates.”

What a wild time to be alive. It’s really quite something. How wonderful it is to have a phrase like “what a wild time to be alive” that could mean a dozen different moderately positive or extremely negative things depending on where in your news or social feed you find this article.

Every Tuesday morning at 7am I have a date.

Abusive traffic from AI-based bots or application is becoming more prevelant which is why I’m thrilled to introduce the general availability of recoil. Recoil is a highly sophisticated honeypot which can serve a never-ending stream of data to abusive traffic.