Databricks announced a new low-latency product powered by a super fast engine they’re calling “Reyden.” The product itself is called Lakehouse//RT and I am both supremely excited to see this in the wild and academically really interested in their approach(es) for implementing Reyden.

Lakehouse//RT is powered by Reyden, a breakthrough new engine for real-time workloads that require immediate responsiveness at high concurrency.

One of the things I have been working on at Scribd has been building a low-latency data retrieval system for arbitrarily large datasets. While not as generally applicable as Reyden, the systems likely share a lot of the same ingredients. I submitted a talk which was not accepted to the event, which makes more sense in hindsight. From my abstract:

Using the same Delta Lake architecture we offer both direct data access for data scientists in Databricks Notebooks and online data retrieval in milliseconds for user-facing web services.

Keynote presentations for me are not typically something I pay attention to, but I knew that Reyden was going to be announced ahead of time and Reynold’s segment was the only one I really paid attention to.

Some points about Reyden which were mentioned are worth revisiting:

  • Databricks used a vast dataset of queries and telemetry from their DBR and Photon engines to train a new model for creating optimized query plans. Using their back catalog of query information, they were able to experiment with different optimization approaches to really tune for different query patterns in novel ways.
  • The architecture behind the scenes was built in such a way to allow for sustaining very high transactions per second while preserving the low latency for all queries.

From their post on the subject:

Preview participants have seen up to 16x better performance vs. real-time serving layers, with response times as low as 10ms on smaller datasets and sub-100ms performance on larger ones. On standard analytical benchmarks, Lakehouse//RT delivers sub-100 millisecond latency at 12,000 queries per second.

Incredible.

I believe that I have an above-average understanding of how hard this actually was, which also makes me proud for Reynold and the team who built his Dream Engine; it is quite an accomplishment.

I wanted to share some wild speculation on what types of techniques could be at play to make Reyden so consistently fast.

What we know

There are a few different ingredients we know play a role in high-throughput low-latency systems like Reyden. Despite not knowing how Reyden was implemented, we can be sure they incorporate some of the same properties we see in other systems:

  • Intelligent query planning: Reynold mentioned as much in his keynote, but one property of every data system is coming up with increasingly intelligent ways to retrieve only the data that is needed. The stuff most people learn about creating indexes in traditional relational data systems gets tilted on its head when building intelligent query planning for large-scale data systems like Reyden. I have seen some systems describe “adaptive query planning” which is flavor of what I believe Reynold discussed: using machine-learning to build models for intelligently improving query plans over time and usage. There is great precedent for this type of work in interpreted languages which I would argue is related. Adaptive query planning reminds me of the work done by Oracle around Graal and Truffle which allowed interpreted code to get faster as runtime went on and the runtime could introspect and adaptively optimize based on real-world usage.

  • Caching: caching is not rocket surgery but the cost of cloud-based RAM was insane before “AI” and it has only gotten worse. Large data-systems must implement tiered caching solutions. Honeycomb has shared some of their work on Retriever which has some novel approaches for storing segments of query results in S3 for reduced retrieval time. Improving the speed of something like Reyden requires consideration of what data to cache but also where to cache it.

  • Object storage: The eighth wonder of the world, as Corey Quinn would like you to believe is S3. The magic sauce that makes some truly incredible data systems possible is commodity object storage with S3. Technologies like Delta Lake and Apache Parquet when wielded appropriately can lead to some pretty incredible low-latency reads.

The Hard Parts

InfluxDB, Lance, Databricks, Scribd, Turbopuffer, and a number of other companies have built and deployed compelling and fast data systems using some of the same underlying technologies that Reyden likely uses. But there are some real limitations that whet my appetite for more details on how exactly did Reyden overcome or hide these realities.

S3 Latency

S3 is the foundational building block of almost anything data in AWS. You can build ridiculously fast data systems only using S3 if you know how to structure your writes. S3 is very fast but single object requests still live in the tens of milliseconds range. Personally I have not seen Standard storage requests below around 20ms.

Each S3 request has some IAM overhead and object routing overhead. This overhead you pay before you get the first byte of data back from the service. The roughly 20ms latency that I have seen is time-to-first-byte, if you need to transfer megabytes of data from S3 to a query/search service the time-to-last byte is the more important metric.

Once you get to the point of streaming bytes, it is possible to stream bytes extremely fast from S3. Multipart objects uploaded with an approximately 8MiB part size will be automatically fetched in parallel by AWS S3 client SDKs. This means that for a 64MiB file, your client can hit multiple backend services simultaneously and stream the parts of that object into memory at what is effectively line-speed of the network interface of the machine.

You can get blazingly fast reads, once you’re reading, but there’s always that 20-30ms latency to begin with.

This sets the floor of latency for a cold request to object storage.

To work around this you must build a multi-tier storage architecture where “interesting data” is being promoted to faster storage like S3 Express OneZone, Aurora/PostgreSQL, Valkey, or other in-memory stores. Meanwhile “uninteresting data” needs to be demoted downward to slower storage, otherwise cost blows up.

Predictive I/O

Databricks groups a couple different techniques under the umbrella of Predictive I/O which in its most simple incantation can be considered “predicate push-down to storage.” I have written about this approach before in Low latency Parquet reads and applied some of these techniques inspired by Andrew and Raphael’s work at Influxdata.

Applying a “predictive I/O” approach in an engine like Reyden is likely extending the “predictiveness” to address a multi-tier storage architecture. I would imagine that underneath the covers there is something along the lines of DuckLake which is denormalizes existing Delta Lake or Iceberg transaction logs into an online storage engine like PostgreSQL. Variants of the work that DuckDB has shared with DuckLake exist elsewhere in the ecosystem and improve performance for high-speed read scenarios with the trade-off of implementation complexity. Inside the walls of Databricks, something like DuckLake could probably be built on top of their Lakebase or Neon to offload that part of the storage/retrieval problem.

Massive distributed compute

The largest most expensive workloads that I have seen on the Databricks platform end up being massive joins across datasets which must all be “considered” by the engine. If the user wants to compare nested JSON values joined between two datasets that are terabytes large, it is difficult to avoid loading or streaming terabytes of data and bringing compute online.

Optimize the I/O and query plan all you want, at some point somebody somewhere will have to do some hard work and actually compute something.

The unsung hero of Reyden is probably the Databricks Photon engine. If you are trying to build something super-fast, starting with a pretty fast engine that’s already in your toolbox is a good place to start.

In my work on a similar but less-generally-applicable system, I have been giving a lot of thought to Apache Datafusion and its ability to serialize logical plans over the wire via Protobuf. I’m not yet sure how I am going to abuse this capability, but I know that it’s a key ingredient in distributing query execution across compute resources in new and novel ways.

In a bespoke cloud environment Databricks has a myriad of ways to distribute compute workloads available to a system designed in 2025/2026. These could operate much more efficiently than Apache Spark operates today, with its driver/worker VM topology. Calling back again to the work that Honeycomb has done, I think they have really demonstrated that extremely bursty compute designs on AWS Lambda are both possible and cost-effective.

Make sure to cheat

Even with a clever query plan, great predictive I/O, and bursty but powerful compute, there are still some laws of physics that have to be respected. Data locality and affinity to compute plays a big role in the speed of data processing engines.

In my own work I have been considering the totality of the stack, which I am sure Reyden does as well. The interaction between a client and server becomes important. What parts of the puzzle are computed on the client versus server? What data is sent back to the client and how?

Imagine you have a very fast engine for your users and they are primarily using it in your Notebook product.

You would be foolish not to *cheat a _little_.

The primary user interface for Reyden is a Databricks notebook. In a vertically integrated stack, it is possible to start preloading data or anticipating user behavior before they actually hit “Run.”

For a totally new notebook, I might start pre-filling caches or moving data into warmer storage as soon as the user has typed a table name in, e.g. once the editor has completed SELECT * FROM gold.schema.table .. I know with high confidence that the user is going to query gold.schema.table and can start to preload.

For an existing notebook that is being revisited, I can cheat slightly differently by preloading the necessary data for all the tables referenced by the cells. If I am feeling very fancy I might rely on a user-behavior-model to anticipate whether they’re going to “Run All”, “Run” one by one, or simply review prior computed results.

Databricks also has been heavily promoting their in-notebook AI Assistant to where I would also cheat a little bit by seeding that AI assistant with instructions on how to write queries that are more efficient on my very fast engine. The AI Assistant should produce code which is going to demonstrate the best possible query patterns for the given user intent.

Unlike general purpose database engines, Reyden has so much other options available to it to give the user the illusion of speed without needing to do as much hard or occasionally impossible work.

“Work smarter, not harder” as the saying goes.

Building a system like Reyden sounds like it was a tremendous amount of fun, with all kinds of unique and interesting constraints. Reyden would not be possible had Databricks not had a strong technology leadership to build from. In that respect both Reynold, Matei, and Ali deserve a tremendous amount of credit for their continued investment in Databricks’ technology platform.

Their investments compound in ways that are difficult to envision in detail “we need to do X, so that we can build Reyden.” The compounding benefits are more easy to imagine in the abstract “making this part of the stack faster/cheaper unlocks new capabilities to build from.”

Delta Lake was needed to move away from CSV, Parquet, Hive, and all manners of other junk holding Apache Spark back from beating Apache Hadoop. Photon was a no-brainer, a fast engine that beats Apache Spark for Databricks users is a foundational component that served an immediate customer need. Lakebase/Neon helps solve a data movement problem faced by every organization with a data lake.

The list of platform investments that are compounding in Reyden is probably much longer than I can imagine. I believe that Reyden could not have existed at Databricks in 2020, 2022, or 2024, but I do think the right pieces of were there in 2025 because of an astute platform-thinking that members of the technology organization have.

From my perspective, I recognize how challenging impressive Reyden is to build, and the long road that a lot of people have been on to make it a reality.

Well done y’all!