Lakehouse//RT is powered by Reyden, a breakthrough new engine for real-time workloads that require immediate responsiveness at high concurrency.

One of the things I have been working on at Scribd has been building a low-latency data retrieval system for arbitrarily large datasets. While not as generally applicable as Reyden, the systems likely share a lot of the same ingredients. I submitted a talk which was not accepted to the event, which makes more sense in hindsight. From my abstract:

Using the same Delta Lake architecture we offer both direct data access for data scientists in Databricks Notebooks and online data retrieval in milliseconds for user-facing web services.

Keynote presentations for me are not typically something I pay attention to, but I knew that Reyden was going to be announced ahead of time and Reynold’s segment was the only one I really paid attention to.

Some points about Reyden which were mentioned are worth revisiting:

• Databricks used a vast dataset of queries and telemetry from their DBR and Photon engines to train a new model for creating optimized query plans. Using their back catalog of query information, they were able to experiment with different optimization approaches to really tune for different query patterns in novel ways.
• The architecture behind the scenes was built in such a way to allow for sustaining very high transactions per second while preserving the low latency for all queries.

From their post on the subject:

Preview participants have seen up to 16x better performance vs. real-time serving layers, with response times as low as 10ms on smaller datasets and sub-100ms performance on larger ones. On standard analytical benchmarks, Lakehouse//RT delivers sub-100 millisecond latency at 12,000 queries per second.

Incredible.

I believe that I have an above-average understanding of how hard this actually was, which also makes me proud for Reynold and the team who built his Dream Engine; it is quite an accomplishment.

I wanted to share some wild speculation on what types of techniques could be at play to make Reyden so consistently fast.

What we know

There are a few different ingredients we know play a role in high-throughput low-latency systems like Reyden. Despite not knowing how Reyden was implemented, we can be sure they incorporate some of the same properties we see in other systems:

• Intelligent query planning: Reynold mentioned as much in his keynote, but one property of every data system is coming up with increasingly intelligent ways to retrieve only the data that is needed. The stuff most people learn about creating indexes in traditional relational data systems gets tilted on its head when building intelligent query planning for large-scale data systems like Reyden. I have seen some systems describe “adaptive query planning” which is flavor of what I believe Reynold discussed: using machine-learning to build models for intelligently improving query plans over time and usage. There is great precedent for this type of work in interpreted languages which I would argue is related. Adaptive query planning reminds me of the work done by Oracle around Graal and Truffle which allowed interpreted code to get faster as runtime went on and the runtime could introspect and adaptively optimize based on real-world usage.

• Caching: caching is not rocket surgery but the cost of cloud-based RAM was insane before “AI” and it has only gotten worse. Large data-systems must implement tiered caching solutions. Honeycomb has shared some of their work on Retriever which has some novel approaches for storing segments of query results in S3 for reduced retrieval time. Improving the speed of something like Reyden requires consideration of what data to cache but also where to cache it.

• Object storage: The eighth wonder of the world, as Corey Quinn would like you to believe is S3. The magic sauce that makes some truly incredible data systems possible is commodity object storage with S3. Technologies like Delta Lake and Apache Parquet when wielded appropriately can lead to some pretty incredible low-latency reads.

The Hard Parts

InfluxDB, Lance, Databricks, Scribd, Turbopuffer, and a number of other companies have built and deployed compelling and fast data systems using some of the same underlying technologies that Reyden likely uses. But there are some real limitations that whet my appetite for more details on how exactly did Reyden overcome or hide these realities.

S3 Latency

S3 is the foundational building block of almost anything data in AWS. You can build ridiculously fast data systems only using S3 if you know how to structure your writes. S3 is very fast but single object requests still live in the tens of milliseconds range. Personally I have not seen Standard storage requests below around 20ms.

Each S3 request has some IAM overhead and object routing overhead. This overhead you pay before you get the first byte of data back from the service. The roughly 20ms latency that I have seen is time-to-first-byte, if you need to transfer megabytes of data from S3 to a query/search service the time-to-last byte is the more important metric.

Once you get to the point of streaming bytes, it is possible to stream bytes extremely fast from S3. Multipart objects uploaded with an approximately 8MiB part size will be automatically fetched in parallel by AWS S3 client SDKs. This means that for a 64MiB file, your client can hit multiple backend services simultaneously and stream the parts of that object into memory at what is effectively line-speed of the network interface of the machine.

You can get blazingly fast reads, once you’re reading, but there’s always that 20-30ms latency to begin with.

This sets the floor of latency for a cold request to object storage.

To work around this you must build a multi-tier storage architecture where “interesting data” is being promoted to faster storage like S3 Express OneZone, Aurora/PostgreSQL, Valkey, or other in-memory stores. Meanwhile “uninteresting data” needs to be demoted downward to slower storage, otherwise cost blows up.

Predictive I/O

Databricks groups a couple different techniques under the umbrella of Predictive I/O which in its most simple incantation can be considered “predicate push-down to storage.” I have written about this approach before in Low latency Parquet reads and applied some of these techniques inspired by Andrew and Raphael’s work at Influxdata.

Applying a “predictive I/O” approach in an engine like Reyden is likely extending the “predictiveness” to address a multi-tier storage architecture. I would imagine that underneath the covers there is something along the lines of DuckLake which is denormalizes existing Delta Lake or Iceberg transaction logs into an online storage engine like PostgreSQL. Variants of the work that DuckDB has shared with DuckLake exist elsewhere in the ecosystem and improve performance for high-speed read scenarios with the trade-off of implementation complexity. Inside the walls of Databricks, something like DuckLake could probably be built on top of their Lakebase or Neon to offload that part of the storage/retrieval problem.

Massive distributed compute

The largest most expensive workloads that I have seen on the Databricks platform end up being massive joins across datasets which must all be “considered” by the engine. If the user wants to compare nested JSON values joined between two datasets that are terabytes large, it is difficult to avoid loading or streaming terabytes of data and bringing compute online.

Optimize the I/O and query plan all you want, at some point somebody somewhere will have to do some hard work and actually compute something.

The unsung hero of Reyden is probably the Databricks Photon engine. If you are trying to build something super-fast, starting with a pretty fast engine that’s already in your toolbox is a good place to start.

In my work on a similar but less-generally-applicable system, I have been giving a lot of thought to Apache Datafusion and its ability to serialize logical plans over the wire via Protobuf. I’m not yet sure how I am going to abuse this capability, but I know that it’s a key ingredient in distributing query execution across compute resources in new and novel ways.

In a bespoke cloud environment Databricks has a myriad of ways to distribute compute workloads available to a system designed in 2025/2026. These could operate much more efficiently than Apache Spark operates today, with its driver/worker VM topology. Calling back again to the work that Honeycomb has done, I think they have really demonstrated that extremely bursty compute designs on AWS Lambda are both possible and cost-effective.

Make sure to cheat

Even with a clever query plan, great predictive I/O, and bursty but powerful compute, there are still some laws of physics that have to be respected. Data locality and affinity to compute plays a big role in the speed of data processing engines.

In my own work I have been considering the totality of the stack, which I am sure Reyden does as well. The interaction between a client and server becomes important. What parts of the puzzle are computed on the client versus server? What data is sent back to the client and how?

Imagine you have a very fast engine for your users and they are primarily using it in your Notebook product.

You would be foolish not to *cheat a _little_.

The primary user interface for Reyden is a Databricks notebook. In a vertically integrated stack, it is possible to start preloading data or anticipating user behavior before they actually hit “Run.”

For a totally new notebook, I might start pre-filling caches or moving data into warmer storage as soon as the user has typed a table name in, e.g. once the editor has completed SELECT * FROM gold.schema.table .. I know with high confidence that the user is going to query gold.schema.table and can start to preload.

For an existing notebook that is being revisited, I can cheat slightly differently by preloading the necessary data for all the tables referenced by the cells. If I am feeling very fancy I might rely on a user-behavior-model to anticipate whether they’re going to “Run All”, “Run” one by one, or simply review prior computed results.

Databricks also has been heavily promoting their in-notebook AI Assistant to where I would also cheat a little bit by seeding that AI assistant with instructions on how to write queries that are more efficient on my very fast engine. The AI Assistant should produce code which is going to demonstrate the best possible query patterns for the given user intent.

Unlike general purpose database engines, Reyden has so much other options available to it to give the user the illusion of speed without needing to do as much hard or occasionally impossible work.

“Work smarter, not harder” as the saying goes.

Building a system like Reyden sounds like it was a tremendous amount of fun, with all kinds of unique and interesting constraints. Reyden would not be possible had Databricks not had a strong technology leadership to build from. In that respect both Reynold, Matei, and Ali deserve a tremendous amount of credit for their continued investment in Databricks’ technology platform.

Their investments compound in ways that are difficult to envision in detail “we need to do X, so that we can build Reyden.” The compounding benefits are more easy to imagine in the abstract “making this part of the stack faster/cheaper unlocks new capabilities to build from.”

Delta Lake was needed to move away from CSV, Parquet, Hive, and all manners of other junk holding Apache Spark back from beating Apache Hadoop. Photon was a no-brainer, a fast engine that beats Apache Spark for Databricks users is a foundational component that served an immediate customer need. Lakebase/Neon helps solve a data movement problem faced by every organization with a data lake.

The list of platform investments that are compounding in Reyden is probably much longer than I can imagine. I believe that Reyden could not have existed at Databricks in 2020, 2022, or 2024, but I do think the right pieces of were there in 2025 because of an astute platform-thinking that members of the technology organization have.

From my perspective, I recognize how challenging impressive Reyden is to build, and the long road that a lot of people have been on to make it a reality.

Well done y’all!

The bad news I have for you is that today, in the year 2026 Unity Catalog does not work with S3 Access Points.

However it does show a different pathology than it once did, which leads me to believe that it could, if not for one silly little piece of technical debt.

The system I am building utilizes Amazon S3 Access Points for governance but must integrate into the Databricks platform. A platform which has its own ideas about governance: Unity Catalog. It should come as no surprise that a system which was named unity would go to great strides to make itself the center of the universe.

How troublesome!

Years ago a colleague and I tried to integrate Databricks Unity Catalog and S3 Access Points only for the approach to crash and burn. Integrating two different opaque tools like IAM permissions and Unity Catalog led to all sorts of attempted incantations, none of which actually succeeded.

The Databricks product team told us that the system did not support S3 Access Points “by design.” I found the reasoning very patronizing because it was presented as “we don’t support S3 Access Points by design to prevent users from circumventing Unity access controls.”

What I understand now is how that “by design” was more of an excuse “we just don’t want to test it” rather than something more substantive.

S3 Access Points can be referenced a number of ways like S3 Access Point Aliases to where even the most legacy system can integrate with them.

An access point alias name meets all the requirements of a valid Amazon S3 bucket name and consists of the following parts:

The first time we bounced off this problem S3 Access Point Aliases had been only recently released;

Despite all Unity Catalog’s protestations the errors we ended up seeing don’t convey a structural limitation when using S3 Access Point Aliases, instead they point to simply out-dated SDK support in the underlying Databricks Runtime.

My hunch is that the AWS SDK v1 being announced as deprecated over two years ago and being completely deprecated as of the end of 2025. Lots of Databricks and other Spark libraries still interact with S3 via the v1 SDK.  That SDK was originally released in 2010 (lol) and so it’s likely that the issue we were authentication issue we were seeing was mixed up in the support for S3 Access Point Aliases with this old SDK.

Since we bounced off this problem a number of years ago one thing that has changed for the better in Unity Catalog is that it is now possible to grant Unity a completely read-only configuration in IAM-based S3 bucket policies. While we cannot use S3 Access Points as part of our governance strategy, we can at least still grant a fairly limited permission to Unity for read-only operations.

Now I can have my esoteric Delta Lake datastores present in Unity without any risk of misconfiguration or error in Unity leading to data corruption!

Governance to a lot of enterprise vendors is about centralization of control, but for me it’s about defense in depth. I never want a business-critical system to be a single misconfiguration away from granting read or write access to the wrong principal.

For years the conventional wisdom around Delta Lake has been to not connect user-facing/online systems to Delta tables. Basically, don’t point your Django app at your Delta tables. This continues to be a decent guideline but definitely not a rule and I have the performance data to back that up.

My talk abstract:

Scribd hosts hundreds of millions of documents and has hundreds of billions of objects across our buckets. Combining large-language models with a massive amounts of text has required investment in our new Content Library architecture.  We selected Delta Lake as the underlying storage technology but have pushed it to an extreme. Using the same Delta Lake architecture we offer both direct data access for data scientists in Databricks Notebooks and online data retrieval in milliseconds for user-facing web services.

In this talk we will review principles of performance for each layer of the stack: web APIs, the Delta Lake tables, Apache Parquet, and AWS S3.

The work done by myself and my colleague Eugene in this area has been heavily related to my previous research around Low latency Parquet reads which informed work named Content Crush, which I have explored more on the Scribd tech blog and on the Screaming in the Cloud podcast.

I really hope that I am able to share results at Data and AI Summit from this incredibly challenging work that I am undertaking. But even if I don’t, blog posts like my musings on Multimodal with Delta Lake, scaling streaming Delta Lake applications, and a myriad of other articles I have published can be pieced together to form the larger mosaic of insane large-scale data work I have been hammering on!

In my opinion Data Engineering Central has been one of the most useful broad-ranged surveys of the ecosystem, curated by one crazy mid-westerner: Daniel. He pulls no punches and while we share criticisms of AI in the industry and commercial tools, Daniel’s honesty also has put some of my work on blast, such as this post about some terrible user-experience and lopsided Delta Lake support in delta-rs.

In his post Daniel highlights some of the topics we got into during our time chatting:

• What the Lakehouse architecture gets right—and where it still falls short
• Why multimodal data (text, images, audio, video, embeddings) changes everything
• How open table formats like Delta Lake fit into the next generation of data platforms
• The growing gap between data tooling hype and day-to-day data engineering reality
• What skills and architectural thinking will matter most for data engineers over the next decade

I encourage you to subscribe to his newsletter or if that’s not your jam, you can subscribe to the RSS feed too.

The delta-rs project started in 2020 and I wish that I could say it is because I am a brilliant visionary. The project largely started because I have had a bias against JVM-based technology stacks and I had stepped into a role at Scribd where we were migrating to AWS, Databricks, and a new architecture anyways so why not challenge the orthodoxy? My colleague QP Hou and I were loving Rust and liked Delta Lake from a design standpoint, but did not love Apache Spark for some of the things we needed to do.

I would consider the official start of the project to be April 11th, 2020 when I sent our Databricks colleagues the following:

Greetings! As I mentioned in our weekly sync up this week, we have an interest in partnering with Databricks to develop and open source native client interface for Delta Lake.

For framing this conversation and scope of the native interface, I categorize our compute workloads into three groups:

1. Big offline data processing, requiring a cluster of compute resources where Spark makes a big dent.
2. Lightweight/small offline data processing, workloads needing “fractional compute” resources, basically less than a single machine. (Ruby/Python type tasks which move data around, or perform small-scale data accesses make up the majority of these in our current infrastructure, we’ve discussed using the Databricks Light runtime for these in the past, since the cost to deploy/run these small tasks on Databricks clusters doesn’t make sense).
3. Boundary data-processing, where the task might involve a little bit of production “online” data and a little bit of warehouse “offline” data to complete its work. In our environment we have Ruby scripts whose sole job is to sync pre-computed (by Spark) offline data into online data stores for the production Rails application, etc, to access and serve.

I don’t want to burn down our current investment in Ruby for many of the 2nd and 3rd workloads, not to mention retraining a number of developers in-house to learn how to effectively use Scala or pySpark.

My proposal is that we partner with Databricks and jointly develop an open source client interface for Delta Lake. One where we would have at least one developer from Databricks working with at least one developer from Scribd on a jointly scoped effort to deliver a library capable of initially addressing our ‘2’ and ‘3’ use-cases.

[..]

Further, I propose that we jointly develop a client interface in Rust, which will allow us to easy extend that within the Databricks community to support Golang, Python, Ruby, and Node clients.

The key benefits I imagine for us all:

• Much broader market share for Delta Lake as a technology. Not only would companies like Scribd benefit, and continue to invest in Delta Lake, but other companies would have an easier on-ramp into the Databricks ecosystem. Basically, if you start using Delta Lake before you use Spark, you will (I guarantee) reach a point where these lightweight workloads become heavyweight workloads requiring the full power and glory of the Databricks runtime :D

• It’s a fantastic developer advocacy story that hits a number of key bullet marketing points: open source, partner collaboration, Rust (so hot right now) :)

• Scribd is able to “immediately” take advantage of Delta Lake benefits without burning up all our existing codebase and investment in Ruby tasks and tooling. Thereby allowing for an easier onramp into Delta Lake and the Databricks platform as a whole.

The scope of the effort I think would be largely around properly dealing with the transaction log, since the Apache Arrow project has already created a pretty decent parquet crate in Rust. That said, there may be some writer improvements we’d want/need to push upstream to Apache Arrow to make this successful.

On second thought, almost all of this has come true! What a brilliant sage! (plz clap)

Like many advancements, there’s a right time, a right place, and a right group of people. Unfortunately Databricks didn’t join the party until a later on but were a strong supporter of our initial work, providing guidance and helping to make Delta Lake an ever-more thriving open source community. The right people were all converging on the direction that made this possible with Neville helped make arrow-rs a much better Apache Parquet writer. QP wrote the first version of the protocol parser and created the first Python bindings for the library. Christian Williams built out kafka-delta-ingest with Mykhailo Osypov and helped prove that: Rust is way more efficient for data ingestion workloads.. As time went on Will Jones, Florian Valeye, and Robert Peck joined the party and helped turn delta-rs from a small Scribd-motivated open source project into a thriving Rust and Python project.

Scribd had wild success with the data ingestion being in Rust, and the data processing/query being in Spark. The community grew, Databricks grew, and at some point some folks started working on a book.

As a long-time maintainer of delta-rs and talking head in the Delta and Databricks ecosystem I was asked to be a technical reviewer of the book after Prashanth, Scott, Tristen, and Denny had already gotten more than halfway through the chapters.

I provided as much feedback as I could on their chapters. I reviewed the outline and noticed “Chapter 8: TBD”.

What’s supposed to be Chapter 8? “We’re not sure yet.”

My friend Kohsuke once marveled at how I was able to acquire things for the Jenkins project by the simple act of asking for them. There’s some skill involved in finding mutually beneficial opportunities, but being uninhibited by the possibility somebody would say “no” helps a lot.

“So this outline looks good, but when are you going to talk about Rust and Python? There are dozens of us! Dozens!”

Denny needed another chapter and I asked if I could write about building native data applications in Rust and Python.

Suddenly I was helping to write a book.

Scribd is a fun company to work at. Books, audiobooks, podcasts, articles. We have a deep appreciation for the written word, telling stories, and learning. All of which I value highly. Before this experience however I had never seen the other side of books. The creation, the meetings, the rewrites, the edits, the reviews, going to press. It is incredibly interesting and the team at O’Reilly are talented, helpful, and professional.

Going through copy-editing I was fielding review comments on the consistency of tense, the subject of sentences, discussions about what is a proper noun and how to consistently apply terms through hundreds of pages of content. I have heard about how invaluable editors are, I have now seen them in action am in awe.

Over the years I have tried and failed to explain what I do to family members. For people that don’t work in tech “working on the computer” all looks largely the same, especially for older generations. Having your work, your name in print has an intangible “wow” factor. More so than conference talks, websites, GitHub stars, or branded t-shirts, a printed artifact recognizes the accomplishments of the innumerable contributors to the Delta Lake ecosystem over the years.

If you’re data inclined, I recommend picking up a copy, Prashanth, Scott, Tristen, and Denny have written a very useful guide, and also I contributed a bit too! :)

Using the excuse of promoting my consulting/professional services company Buoyant Data I had effectively three speaking engagements:

• The road to delta-rs 1.0 at the Open Source Contributor Summit (Monday)
• Fast, cheap, and easy data ingestion with AWS Lambda and Delta Lake, a talk highlighting a lot of the successful patterns I have developed for customers using AWS Lambda with Delta Lake for Rust to create shockingly cheap data ingestion pipelines. (Thursday)
• Let’s do data engineering in Rust!, a more fun deep-dive talk to help people start to get into the world of implementing data systems with Rust. (Thursday)(

Unfortunately the first talk was not recorded, but it was probably the most interesting! On Monday morning I was riding my bike from the Ferry Building to the venue in San Francisco and my chain snapped off while I was sprinting off from a green light. I went down hard, scraped up my knees, and generally looked a fool lying in the middle of Market St.

The show must go on, so I hobbled to the Scribd office, deposited my broken bike, and continued to the Open Source Summit.

What I did not know at the time was that I had fractured a bone in my wrist. I did know however that I needed to go to a clinic, but really wanted to attend the summit and take advantage of the one-a-year opportunity (literally!) for some of the brightest minds in the data community to talk about the future of Delta Lake and more.

So that first talk was given with my swollen wrist pulled to my heart, like a broken wing, and I’m sure it was a ludicrous sight to see!

By Thursday my arm had been set and was in a sling, which is far less exciting. Nonetheless, the two talks below are perhaps the only one-handed presentations thus far in my career! I hope you enjoy!

Note: The presentation software used for this talk is the open source presenterm tool which is delightful for creating development-focused presentations like this one!

Like many pieces of technology, it is not inherently good or bad, but the problem with ML as it is applied today is that its application is far outpacing our understanding of its consequences.

Brian Kernighan, co-creator of the C programming language and UNIX, said:

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

Setting aside the mountain of ethical concerns around the application of ML which have and should continue to be discussed in the technology industry, there’s a fundamental challenge with ML-based systems: I don’t think their creators understand how they work, how their conclusions are determined, or how to consistently improve them over time. Imagine you are a data scientist or ML developer, how confident are you in what your models will predict between experiments or evolutions of the model? Would you be willing to testify in a court of law about the veracity of your model’s output?

Imagine you are a developer working on the models that Tesla’s “full self-driving” (FSD) mode relies upon. Your model has been implicated in a Tesla killing the driver and/or pedestrians (which has happened). Do you think it would be possible to convince a judge and jury that your model is not programmed to mow down pedestrians outside of a crosswalk? How do you prove what a model is or is not supposed to do given never before seen inputs?

Traditional software does have a variation of this problem but source code lends itself to scrutiny far better than the ML models. Many of which have come from successive evolutions of public training data, proprietary model changes, and integrations with new data sources.

These problems may be solvable in the ML ecosystem, but problem is that the application of ML is outpacing our ability to understand, monitor, and diagnose models when they do harm.

That model your startup is working on to help accelerate home loan approvals based on historical mortgages, how do you assert that your models are not re-introducing racist policies like redlining. (forms of this have happened).

How about that fun image generation (AI art!) project you have been tinkering with uses a publicly available model that was trained on millions of images from the internet, and as a result in some cases unintentionally outputs explicit images, or even what some jurisdictions might consider bordering on child pornography. (forms of this have happened).

Really anything you teach based on the data “from the internet” is asking for racist, pornographic, or otherwise offensive results, as the Microsoft Tay example should have taught us.

Can you imagine the human-rights nightmare that could ensue from shoddy ML models being brought into a healthcare setting? Law-enforcement? Or even military settings?

Machine-learning encompasses a very powerful set of tools and patterns, but our ability to predict how those models will be used, what they will output, or how to prevent negative outcomes are dangerously insufficient for the use outside of search and recommendation systems.

I understand how models are developed, how they are utilized, and what I think they’re supposed to do.

Fundamentally the challenge with AI/ML is that we understand how to “make it work”, but we don’t understand why it works.

Nonetheless we keep deploying “AI” anywhere there’s funding, consequences be damned.

And that’s a problem.

Mike Julian from The Duckbill Group once made the proclamation that the way to actually save money in AWS is to design your infrastructure to be cost-effective. “Optimization” techniques can only take you so far, and once you’ve burned through all the optimizations, you may find yourself needing to further reduce the cost of your infrastructure and have no more “fat” to trim! In the first blog post I outline a “reference architecture” for a data platform which I know is cost-effective, easy to manage, and lends itself well to growth.

Planning for sensible, cost-concious growth is very important. With most data platforms as they start to prove their value, the organization will bring even more workloads to them. If you give a data scientist a good platform, they will find themselves wanting ever more from that data platform, and Buoyant Data can help make sure that growth is sustainable and the value to the business is easy to identify as well.

Please add the Buoyant Data RSS feed to your reader, as I have a number of blog posts queued up already with some gratis tips and tricks for understanding the cost of your data platform! 😄

The technology stack for Buoyant Data is something I cannot wait to write more about. After funding the creation of delta-rs as part of my day job, I am utilizing the library in a big way to build extremely lightweight and cost-efficient data ingestion pipelines with Rust and AWS Lambda. There’s still plenty of space for Apache Spark on the querying and processing side, but as DataFusion matures, I’m looking forward to exploring where that can fit into the picture.

There’s a lot of evolution happening right now in the data and ML platform space, I’m really looking forward to growing Buoyant Data in my spare time!

My originating data set was a comma-separated values file (CSV) so my first intent was to just load it into SQLite using the .mode csv command in the CLI, but I found that to be a bit restrictive. Notebooks have incredible utility for incrementally working on data. Unfortunately Jupyter doesn’t have a native SQL interface, instead everything has to run through Python. Through my work with delta-rs I am somewhat familar with Pandas for processing data in Python, so my first attempts where using the Pandas data frame API to munge through my data.

import pandas

df = pandas.read_csv('data/2021_05-2022_04.csv')

I could be dense, but I find SQL to be a pretty understandable tool in comparison to data frames, so I needed to find some way to get the data into a SQL interface. The solution that I ended up with was to create an in-memory SQLite database and use Pandas to query it, which works okay enough to where I continued working and didn’t bother thinking too much about how to optimize the approach further:

import sqlite3
import pandas

# Loading everything into a SQLite memory database because I hate data frames and SQL is nice
conn = sqlite3.connect(':memory:')
df = pandas.read_csv('data/2021_05-2022_04.csv')
r = df.to_sql('usage', conn, if_exists='replace', index=False)
# useful little helper
sql = lambda x: pandas.read_sql_query(x, conn)


# Show some sample data
sql('SELECT * FROM usage LIMIT 3')

The benefit of this approach is that I can create additional tables in the SQLite database with static data sets, or other CSVs. Since I’m also just doing some simple ad-hoc analysis, I can skip writing anything to disk and keep things snappy in memory.

I created the little sql lambda to make the notebook a bit more understandable, and to get out of exposing the cursor or database connection to every single cell, meaning that most of my cells in the notebook are simply just sql('SELECT * FROM foo') statments with some documentation surrounding them.

Fairly simple, easy enough to play with data quickly on my local machine without invoking all the infinite cosmic powers the cloud provides!

From the site itself:

The Databricks Beacons program is our way to thank and recognize the community members, data scientists, data engineers, developers and open source enthusiasts who go above and beyond to uplift the data and AI community.

Whether they are speaking at conferences, leading workshops, teaching, mentoring, blogging, writing books, creating tutorials, offering support in forums or organizing meetups, they inspire others and encourage knowledge sharing – all while helping to solve tough data problems.

I’m flattered to be included in the inaugural group of Beacons, which include a number of much more competent data leaders than myself. Most of what I bring to the table is a lot of Delta Lake experience and advocacy. Delta Lake is the bedrock of Scribd’s data platform and I have been investing heavily in the space with our contribution of the delta-rs Rust bindings, upon which kafka-delta-ingest was built.

Scribd is a Databricks customer, and from that angle I have been quite impressed with the organization and technologies they have built. As some folks who have seen my public talks about Databricks, I also don’t hold back in my honest assessment of the platform’s strengths and weaknesses, thus my surprise to be included as a Beacon ;)

I’m looking forward to more events where I am able to share some of the real-world experiences we’re gaining at Scribd in building out massive data platform systems with Delta Lake and Databricks. And as always, if you want to help us build out more feel free to email me!