Want to create an interactive transcript for this episode?
Podcast: Developer Voices
Episode: Batch Data & Streaming Data in one Atom (with Jove Zhong)
Description: Every database has to juggle the need to process new data and to query old data. That task falls to any system that “does stuff and remembers stuff”. But it’s quite hard to really optimise one system for both use cases. There are different constraints on new and old data, and as a system gets larger and larger, those differences multiply to breaking point. That’s something Twitter’s engineers were figuring out in the 2010s.One solution that came up in those years was the Lambda Architecture. A two-pronged approach that recognises the divide between new and ol...