Medallion Architecture Azure End to End Data Engineering Project | Real World Data Solution

In this video, we walk through a complete Azure Data Engineering Project from start to finish. We begin by moving on-premises data to the Azure Cloud using Azure Data Factory with a Self-hosted Integration Runtime. Once the data is in Data Lake Storage, we perform data transformations in Azure Databricks, securely connected via Azure Key Vault. Finally, we visualize the cleaned and structured data by building an interactive Power BI dashboard. Throughout the project we implement Medallion Architecture to ensure efficient data processing and management in the project. This end-to-end pipeline covers real-world Azure services and is perfect for your portfolio or interview preparation. Technologies Used: 1. Azure Data Factory 2. Self-hosted Integration Runtime 3. Azure Data Lake Storage Gen2 4. Azure Databricks 5. Azure Key Vault 6. Power BI Timecodes 0:00 - Coming up 0:30 - Introduction 1:40 - Architecture Diagram 3:48 - Downloading dataset from Maven Analytics 7:12 - Create an Azure Resource Group 8:45 - Create an Azure Storage account 16:30 - Create an Azure Data Factory 19:18 - How to move on-premises data to Azure cloud using Self Hosted Integration Runtime (SHIR) 26:10 - Creating Pipeline in Azure Data Factory 41:30 - Recap 43:08 - How to connect Azure Databricks with Azure Storage Account 49:38 - How to create an Azure Databricks scope 50:50 - Create an Azure Key Vault 54:35: Create an Azure Key Vault Secret 56:50 - How to create an Azure Databricks Cluster 59:30 - Transformations in Azure Databricks 1:21:00 - How to move transformed data in Azure Databricks to Azure Storage Account 1:42:30 - How to connect Power BI to Azure Storage account using Account keys 1:52:50 - Developing a dashboard in Power BI If you like the video hit the like button, and subscribe the channel. I will see you in the next video. Music track: Careful by Pufino Source: https://freetouse.com/music Royalty Free Music for Videos (Safe) Cylinder Five by Chris Zabriskie is licensed under a Creative Commons Attribution 4.0 license. https://creativecommons.org/licenses/by/4.0/ Source: http://chriszabriskie.com/cylinders/ Artist: http://chriszabriskie.com/