[WITH Azure DevOps] Azure End-To-End Data Engineering Project 2025 (FROM SCRATCH!)

Azure End-To-End Data Engineering Project | Azure DevOps | Azure Data Factory | Databricks | PySpark In this video, I guide you through a beginner-friendly, end-to-end Azure Data Engineering project using the latest technologies like Azure Data Factory, Databricks, PySpark, Azure Data Lake, and CI/CD with Azure DevOps. Plus, I cover some Azure Data Engineer interview questions! Timestamps: 0:00 Introduction 08:41 Azure Project Architecture 15:31 Data Understanding 22:04 Azure Free Account 26:08 Azure Fundamentals 32:23 Creating Azure Data Lake 42:26 CI CD in Azure 57:46 Azure DevOps Tutorial 1:07:57 Azure Data Factory Git Configuration 1:17:03 Data Ingestion from Github in Azure Data Factory 1:30:56 Parametrized ETL Pipelines using ForEach Activity 2:01:52 Azure Data Factory Real Time Scenarios 2:32:07 Creating Pull Request in Azure DevOps 2:40:42 ARM templates in Azure 2:45:00 Azure Databricks Tutorial 3:02:00 Databricks with Unity Catalog 3:20:54 Data Ingestion using PySpark in Databricks 3:42:00 Dynamic Notebooks using Databricks Utilities 3:58:11 Data Orchestration using Databricks Workflows 4:16:12 PySpark Tutorial 4:33:04 Big Data Transformation using PySpark 4:56:43 Databricks Delta Live Tables 5:11:27 Spark Structured Streaming 5:26:32 Change Data Capture (CDC) in Databricks 5:36:30 Apache Spark Cluster 5:38:56 End To End ETL Pipeline in Databricks WATCH MY PREVIOUS VIDEOS ➡️Azure End To End Data Project https://youtu.be/uc-u_juRg-w?si=sO_CavjnhNaSnZri ➡️Databricks Tutorial https://youtu.be/7pee6_Sq3VY?si=pPBrTnE2R8PhhHu5 ➡️PySpark Full Course https://youtu.be/94w6hPk7nkM?si=i3u_8ZWkGf6fiGyh REPOSITORY ➡️GitHub https://github.com/anshlambagit/AzureProjectWithCICD ➡️Original Dataset https://www.kaggle.com/datasets/piterfm/paris-2024-olympic-summer-games ======================================= Join this channel to SUPPORT MY HARDWORK: https://www.youtube.com/channel/UCu7lQE-L5gzt8aD7zuuufjw/join ======================================= Connect with ME LinkedIn - https://www.linkedin.com/in/ansh-lamba-793681184/ Telegram - https://t.me/anshlambadatafam ========================= For COLLABORATION 👇 [email protected] ========================= ⭐Hashtags⭐ #azure #databricks #dataengineering #pyspark #azuredatafactory