Architecting the Snowflake Data Cloud by Allison Lee (Dijkstra Award 2024)

Speaker: Allison Lee Abstract: In 2014, Snowflake emerged from stealth mode and announced a "data warehouse, built for the cloud". The system was built from the ground up, using a new architecture called the "multi-cluster shared data" architecture, which separates compute from storage, in order to allow users to take advantage of the elasticity of the cloud. In the decade since then, our system has evolved into a global, multi-cloud platform that supports data collaboration and many data processing workloads beyond data warehousing. In this talk, I will explain why the multi-cluster shared data architecture is well suited for the cloud, and how this architecture has enabled capabilities like massive scalability, automatic optimizations, and global secure data collaboration. Finally, I'll outline how the architecture has evolved over time as our capabilities have grown, and a few lessons learned along the way. Biography: Allison Lee has spent the last two decades building commercial database systems. She currently leads the database engineering team at Snowflake, which owns the core query processing stack, Snowflake's internal key-value metadata store, and the data warehousing and hybrid transactional-analytic workloads. Allison joined Snowflake in 2013 as a founding engineer, and contributed to many of Snowflake's foundational components, including the query optimizer, metadata management, language extensibility, and data protection. Prior to Snowflake, Allison worked on the Oracle database, with a focus on query optimizations for analytical workloads and bridging the gap between query optimization and execution with adaptive optimization techniques. Allison holds dozens of patents in query optimization, data protection, and secure data collaboration. Allison holds a Bachelor of Science and a Master of Engineering in electrical engineering and computer science from the Massachusetts Institute of Technology. Website: https://www.cwi.nl/en/events/dijkstra-awards/cwi-lectures-dijkstra-fellowship/ About the Dijkstra Fellowship The Dijkstra Fellowship is named after former CWI researcher Edsger W. Dijkstra, who was one of the most influential scientists in the history of CWI. Dijkstra developed the shortest path algorithm, among other contributions. The first Dijkstra Fellowships were awarded to David Chaum and Guido van Rossum in 2019. Dijkstra Fellowship 2024 for Marcin Żukowski Marcin Żukowski started his career at CWI. He did his MSc and PhD research on database management system architectures in our Database Architectures (DA) group. As a PhD student under the supervision of Peter Boncz, he developed the innovative concept of vectorized execution to improve the performance of database queries. This research received the DaMoN 2007 Best Paper Award and also the CIDR 2024 Test of Time Award, established by the Conference on Innovative Data Systems Research (CIDR).