Microsoft Fabric: Lakehouse & Medallion Architecture - Part 1

Microsoft Fabric End to End Demo - Part 1 - Lakehouse & Medallion Architecture. In this video we look at the #Lakehouse and #MedallionArchitecture concepts, and how they relate to #MicrosoftFabric. Lakehouses have become a common foundational design pattern for modern data platforms, and have risen in popularity over the last few years after tools like #Databricks, and now #MicrosoftFabric, have standardized on the design. The Medallion Architecture refers to a three-tiered architectural blueprint, with data moving through different stages of state, each stage serving different purposes. 00:00 Intro 00:22 What is a Lakehouse? 02:10 What is a Lakehouse in Microsoft Fabric? 04:43 Table and File structure in OneLake 05:19 Justification for Medallion architecture 06:13 Medallion architecture overview 07:00 Bronze layer 08:15 Silver layer 08:45 Gold layer 09:50 Replayability & idempotency 10:56 Alternative structures 11:52 Outro Useful links: 📖 Lakehouse MSFT docs: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-overview 📖 Medallion architecture MSFT docs: https://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion Series contents: 📺 Part 1 - Lakehouse & Medallion Architecture - https://www.youtube.com/watch?v=x_CvCwSbRZI&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=1&ab_channel=endjin 📺 Part 2 - Planning and Architecting a Data Project - https://www.youtube.com/watch?v=miDa4FZY7GU&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=2&ab_channel=endjin 📺 Part 3 - Ingest Data - https://www.youtube.com/watch?v=yptYlWLsVYk&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=3&ab_channel=endjin 📺 Part 4 - Creating a shortcut to ADLS Gen2 in Fabric - https://www.youtube.com/watch?v=11urrDAW-bU&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=4&ab_channel=endjin 📺 Part 5 - Navigating OneLake data locally - https://youtu.be/DO779ZcAtTk 📺 Part 6 - Role of the Silver layer in the Medallion Architecture - https://www.youtube.com/watch?v=pCTl-nqDT_8&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=6&ab_channel=endjin 📺 Part 7 - Processing Bronze to Silver - https://www.youtube.com/watch?v=s_mHaLBlA94&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=7&ab_channel=endjin 📺 Part 8 - Good Notebook Development Practices - https://www.youtube.com/watch?v=UyS6ZUgh-Wc&list=PLJt9xcgQpM61fxyB1aWzWCAEsHZHEZD6w&index=7&ab_channel=endjin If you want to learn more about Fabric, take a look at some of our other content: 🤖 [Course] Microsoft Fabric: from Descriptive to Predictive Analytics: https://www.youtube.com/watch?v=uaRePHeqvQU&list=PLJt9xcgQpM63xH0x9K-oQMFiE4D9pdl4W 👉 Perspectives on #MicrosoftFabric: https://endjin.com/what-we-think/talks/perspectives-on-microsoft-fabric 👉 A Tour Around #MicrosoftFabric: https://endjin.com/what-we-think/talks/a-tour-around-microsoft-fabric 👉 Introduction to #MicrosoftFabric: https://endjin.com/blog/2023/05/intro-to-microsoft-fabric and find all the rest of our content here: https://endjin.com/blog/2023/05/microsoft-fabric-announced. #Microsoft #PowerBI #MicrosoftFabric #lakehouse #datalake #onelake #databricks #ai #analytics #medallion #bronze #silver #gold #synapse #azuresynapse