I was lucky enough to record a podcast episode with Daniel Beach of Data Engineering Central. Daniel and I have known each other for a couple years sharing notes and ideas on the state of the ecosystem, where it falls down, and where things are getting interesting.
In my opinion Data Engineering Central has been one of the most useful broad-ranged surveys of the ecosystem, curated by one crazy mid-westerner: Daniel. He pulls no punches and while we share criticisms of AI in the industry and commercial tools, Daniel’s honesty also has put some of my work on blast, such as this post about some terrible user-experience and lopsided Delta Lake support in delta-rs.
In his post Daniel highlights some of the topics we got into during our time chatting:
- What the Lakehouse architecture gets right—and where it still falls short
- Why multimodal data (text, images, audio, video, embeddings) changes everything
- How open table formats like Delta Lake fit into the next generation of data platforms
- The growing gap between data tooling hype and day-to-day data engineering reality
- What skills and architectural thinking will matter most for data engineers over the next decade
I encourage you to subscribe to his newsletter or if that’s not your jam, you can subscribe to the RSS feed too.