Building a new Database Query Optimiser - @cmu ​

Read more about Kafka Diskless-topics, KIP by Aiven: KIP-1150: https://fnf.dev/3EuL7mv Summary: In this conversation, Kaivalya Apte and Alexis Schlomer discuss the internals of query optimization with the new project optd. They explore the challenges faced by existing query optimizers, the importance of cost models, and the advantages of using Rust for performance and safety. The discussion also covers the innovative streaming model of query execution, feedback mechanisms for refining optimizations, and the future developments planned for optd, including support for various databases and enhanced cost models. Chapters 00:00 Introduction to optd and Its Purpose 03:57 Understanding Query Optimization and Its Importance 10:26 Defining Query Optimization and Its Challenges 17:32 Exploring the Limitations of Existing Optimizers 21:39 The Role of Calcite in Query Optimization 26:54 The Need for a Domain-Specific Language 40:10 Advantages of Using Rust for optd 44:37 High-Level Overview of optd's Functionality 48:36 Optimizing Query Execution with Coroutines 50:03 Streaming Model for Query Optimization 51:36 Client Interaction and Feedback Mechanism 54:18 Adaptive Decision Making in Query Execution 54:56 Persistent Memoization for Enhanced Performance 57:12 Guided Scheduling in Query Optimization 59:55 Balancing Execution Time and Optimization 01:01:43 Understanding Cost Models in Query Optimization 01:04:22 Exploring Storage Solutions for Query Optimization 01:07:13 Enhancing Observability and Caching Mechanisms 01:07:44 Future Optimizations and System Improvements 01:18:02 Challenges in Query Optimization Development 01:20:33 Upcoming Features and Roadmap for optd References: - NeuroCard: learned Cardinality Estimation: https://vldb.org/pvldb/vol14/p61-yang.pdf - RL-based QO: https://arxiv.org/pdf/1808.03196 - Microsoft book about QO: https://www.microsoft.com/en-us/research/publication/extensible-query-optimizers-in-practice/ - Cascades paper: https://15721.courses.cs.cmu.edu/spring2016/papers/graefe-ieee1995.pdf - optd source code: https://github.com/cmu-db/optd - optd website (for now): https://db.cs.cmu.edu/projects/optd/ For memberships: join this channel as a member here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join Don't forget to like, share, and subscribe for more insights! ============================================================================= Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator ============================================================================= Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #database #queryoptimization #sql #postgres