Azure End-To-End Data Engineering Project for Beginners (FREE Account) | SQL DB Tutorial

💻 Beta test my AI App: https://forms.gle/tZ3mFxHEScktUwfq8 🤖 Early access to the Applied AI Community: https://kickofflabs.com/waitlist/f795d9da Get the resources (Github) 👉 https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo Join the Community 👉 https://www.youtube.com/channel/UCoaAe6ighxrzJuHeFPAlCeA/join This project addresses a critical business need by building a comprehensive data pipeline on Azure. The goal is to extract customer and sales data from an on-premises SQL database, transform it in the cloud, and generate actionable insights through a Power BI dashboard. The dashboard will highlight key performance indicators (KPIs) related to gender distribution and product category sales, allowing stakeholders to filter and analyze data by date, product category, and gender. **Tech Stack:** On-Prem SQL DB, Data Factory, Data Lake Gen 2, Databricks, Synapse Analytics, Power BI, Entra ID (Active Directory), Key Vault # Watch Next https://youtu.be/lyp8rlpJc3k https://youtu.be/P7EqW6_7wKs https://youtu.be/gpz6YTnSSGY https://youtu.be/pfgOuduvYmQ # Courses & Certificates 👇 Check them out for free 👇 ## Data Engineer 🌍 Understanding Data Engineering 👉 https://datacamp.pxf.io/JKq0ZE 🏅 Database Design 👉 https://datacamp.pxf.io/Xm1AEg 📊 Data Warehousing Concepts 👉 https://datacamp.pxf.io/WyYaPX ⚡ Introduction to dbt 👉 https://datacamp.pxf.io/09Y7BJ 🕹️ Introduction to Apache Airflow in Python 👉 https://datacamp.pxf.io/nXn3eX 📈 Introduction to NoSQL 👉 https://datacamp.pxf.io/6yG9RG ## Cloud Engineer 🌤️ Understanding Cloud Computing 👉 https://datacamp.pxf.io/LKDqYY 1️⃣ Understanding Microsoft Azure 👉 https://datacamp.pxf.io/mOn3k1 2️⃣ AWS Concepts 👉 https://datacamp.pxf.io/7abrBQ 📊 Microsoft Azure Architecture and Services 👉 https://datacamp.pxf.io/MAQxYo ⚙️ Microsoft Azure Management and Governance 👉 https://datacamp.pxf.io/YRdGkj 🧩 AWS Cloud Technology and Services Concepts 👉 https://datacamp.pxf.io/4GdDBo # Social Media 📧 Newsletter: https://lukejbyrne.com/subscribe 🌄 Instagram: https://www.instagram.com/dataluke_ ⏰ TikTok: https://www.tiktok.com/@dataluke 📚 Linkedin: https://linkedin.com/in/lukejbyrne 🛍️ Amazon storefront: https://www.amazon.com/shop/lukejbyrne 🏆 1:1 Coaching: https://calendly.com/lukejbyrne/30min Some links included are affiliate links, which help me keep this channel going. Thanks for the support. Business inquiries: [email protected] Further detail on setting up SSMS and SQL: https://youtu.be/z7o5Wju-PZg?si=QnMF0AVB5DxNf182 How to set up PowerBI without work email: https://www.youtube.com/watch?v=9RB5xic9BiY Mr Ks original vid: https://youtu.be/iQ41WqhHglk?si=drPsOQDhLy-gPIbN Overview --- 00:00:00 - Introduction 00:01:18 - Setting Up the Azure Environment 00:05:45 - SQL Database Configuration 00:10:30 - Overview of Azure Data Lake Storage SSMS --- 00:15:23 - Configuring Azure Data Factory 00:25:11 - Copying Data from SQL to Data Lake 00:38:05 - Debugging Initial Pipeline Issues Azure Data Factory --- 00:45:13 - ForEach Activity in Azure Data Factory 00:55:30 - Testing the SQL-to-Bronze Pipeline 01:05:30 - Recap of SQL-to-Bronze Process 01:08:41 - Debugging the Pipeline 01:10:04 - Monitoring Pipeline Runs 01:10:28 - Verifying Data in Bronze Layer 01:11:14 - Completion of the Bronze Data Layer Databricks --- 01:11:53 - Starting Databricks Configuration 01:14:43 - Creating a Databricks Cluster 01:17:29 - Mounting Data Lake Storage in Databricks 01:23:00 - Transformation in Databricks (Bronze to Silver) 01:33:06 - Automating Data Transformations 01:37:03 - Integrating Databricks with Data Factory 01:41:33 - Pipeline Testing and Monitoring Synapse Analytics --- 01:45:25 - Loading Data into Synapse Analytics 01:50:07 - Creating Views in Synapse 01:54:40 - Integrating Synapse Views into Data Factory Pipelines Power BI --- 01:57:57 - Power BI Dashboard Setup 02:03:11 - Building Relationships in Power BI 02:06:48 - Dashboard Filters and Slicers 02:10:01 - Publishing and Sharing Power BI Dashboards Automation and Active Directory --- 02:13:03 - Automating the Entire Pipeline 02:17:11 - Active Directory (Entra ID) Integration 02:21:33 - Triggering and Monitoring Automated Pipelines 02:29:43 - Final Dashboard Refresh and Validation Closing --- 02:30:07 - Closing Remarks and Next Steps