End-to-End Azure ETL | SQL➡️ADLS with ADF, Databricks, Synapse & Power BI | Beginner + Intermediate

🚀 Welcome to DataToCrunch! In this video, we’re building a complete end-to-end ETL pipeline using Microsoft Azure — from connecting to an on-premise SQL Server all the way to creating stunning Power BI dashboards! 👨‍💼 Imagine working for a company where data lives on-prem, but your analytics team is in the cloud. How do you securely and efficiently move that data, clean it, model it, and visualize it? This project has you covered — step by step. 🔧 What You’ll Learn: ✅ Restore & explore the AdventureWorks2019 SQL Server database ✅ Connect ADF to on-prem SQL using Self-hosted Integration Runtime ✅ Use Azure Data Factory to dynamically copy multiple tables ✅ Store data in ADLS Gen2 using the Bronze, Silver, Gold layers ✅ Use Azure Key Vault for secure secret management ✅ Transform data using Azure Databricks ✅ Views & External Tables creation using Azure Synapse Analytics ✅ Create stunning dashboards in Power BI 🔧Tech Stack Used: ✅ SQL Server 2019 + SSMS ✅ Azure Data Factory (ADF) ✅ Azure Data Lake Storage Gen2 (ADLS) ✅ Azure Key Vault ✅ Azure Databricks ✅ Azure Synapse Analytics ✅ Power BI 🔧 Time Stamp: 00:00:00 - Intro 00:01:21 - A] Dataset Understanding 00:02:51 - B] Project WorkFlow 00:07:02 - C] On-Premise SQL Server & SSMS Installations 00:11:19 - D] SSMS - Restore Database 00:15:38 - E] Management Level & Hierarchy Concept 00:20:29 - 1) Subscription Creation 00:21:23 - 2) Resource Groups Creation 00:23:20 - 3) Azure Data Lake Storage Services Creation 00:27:43 - F] Understanding Azure Storage Redundancy Offerings 00:44:09 - 4) Azure Key Vault Creation 00:50:24 - 5) Azure Data Factory Creation 00:56:24 - 6) ADF WorkFlow 01:05:00 - 7) ADF - Copy Activity 01:24:08 - 8) ADF - LookUp + For Each + Copy Activity 02:03:42 - 9) Azure Databricks Creation 02:07:27 - 10) ADB - Compute Creation 02:15:04 - 11) ADB - Silver Layer Work - Data Cleaning & Transformation 03:02:41 - 12) Azure Synapse Analytics Creation 03:22:34 - 13) Azure Synapse Analytics - Gold Layer - Practical Implementation 04:03:32 - 14) Connecting to Power BI 04:11:20 - 15) Conclusion ✅GitHub Link : https://github.com/RutujaKadam95/End-to-End-Azure-ETL-SQL-ADLS-with-ADF-Databricks-Synapse-Power-BI-Beginner-Intermediate ✅Dataset : https://learn.microsoft.com/en-us/sql/samples/adventureworks-install-configure?view=sql-server-ver16&tabs=ssms ✅Download & Install SQL Server : https://www.microsoft.com/en-ca/sql-server/sql-server-downloads ✅SSMS : Download SQL Server Management Studio (SSMS) | Microsoft Learn ✅Azure Portal : https://azure.microsoft.com/en-ca/get-started/azure-portal 🔔 Don’t forget to Like, Subscribe, and hit the notification bell for more tutorials on Data Engineering, ETL, Azure, and Big Data! 🧠Till then, keep crunching the data! — by DataToCrunch. #azure #azuredataengineer #etl #Data #DataPipelines #DataAnalyses #DataIntegration #BusinessIntelligence #ETL #sqlserver #ssms #azuredatafactory #azuredatabricks #adls #azuresynapse #powerbi #realtime #datalake