The State & Future of Apache Kafka (with Anatoly Zelenin)

I’m joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It’s an approach (and a whole market) that’s had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What’s still left to build? – Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-action Pat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014 Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdf Martin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html Data Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398 Quix Streams: https://github.com/quixio/quix-streams XTDB: https://xtdb.com/ -- Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@DeveloperVoices/join Anatoly’s Website: https://zelenin.de/ Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins -- 0:00 Intro 3:38 What Is Apache Kafka? 10:52 Is Kafka A Database? 15:31 How Are Companies Adopting Kafka? 18:25 Is The Industry Dragging Its Feet On Async? 24:07 Inside Data vs Outside Data 30:48 Turning the Database Inside-Out 40:37 What Does The Future Of Kafka Look Like? 56:31 Disparate Teams and Data Meshes 1:07:51 What's the Use of Kafka Today? 1:11:16 Outro