Databricks Data Engineering Project for Beginners (FREE Account) | Azure Tutorial

💻 Beta test my AI App: https://forms.gle/tZ3mFxHEScktUwfq8 🤖 Early access to the Applied AI Community: https://kickofflabs.com/waitlist/f795d9da Get the resources (Github) 👉 https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo Join the Community 👉 https://www.youtube.com/channel/UCoaAe6ighxrzJuHeFPAlCeA/join Let's make an End-to-End Data Pipeline in Databricks. # Watch Next https://youtu.be/lyp8rlpJc3k https://youtu.be/P7EqW6_7wKs https://youtu.be/ygJ11fzq_ik https://youtu.be/gpz6YTnSSGY # Courses & Certificates 👇 Check them out for free 👇 ## Data Engineer 🌍 Understanding Data Engineering 👉 https://datacamp.pxf.io/JKq0ZE 🏅 Database Design 👉 https://datacamp.pxf.io/Xm1AEg 📊 Data Warehousing Concepts 👉 https://datacamp.pxf.io/WyYaPX ⚡ Introduction to dbt 👉 https://datacamp.pxf.io/09Y7BJ 🕹️ Introduction to Apache Airflow in Python 👉 https://datacamp.pxf.io/nXn3eX 📈 Introduction to NoSQL 👉 https://datacamp.pxf.io/6yG9RG ## Cloud Engineer 🌤️ Understanding Cloud Computing 👉 https://datacamp.pxf.io/LKDqYY 1️⃣ Understanding Microsoft Azure 👉 https://datacamp.pxf.io/mOn3k1 2️⃣ AWS Concepts 👉 https://datacamp.pxf.io/7abrBQ 📊 Microsoft Azure Architecture and Services 👉 https://datacamp.pxf.io/MAQxYo ⚙️ Microsoft Azure Management and Governance 👉 https://datacamp.pxf.io/YRdGkj 🧩 AWS Cloud Technology and Services Concepts 👉 https://datacamp.pxf.io/4GdDBo # Social Media 📧 Newsletter: https://lukejbyrne.com/subscribe 🌄 Instagram: https://www.instagram.com/dataluke_ ⏰ TikTok: https://www.tiktok.com/@dataluke 📚 Linkedin: https://linkedin.com/in/lukejbyrne 🛍️ Amazon storefront: https://www.amazon.com/shop/lukejbyrne 🏆 1:1 Coaching: https://calendly.com/lukejbyrne/30min Some links included are affiliate links, which help me keep this channel going. Thanks for the support. Business inquiries: [email protected] 0:00 - Introduction 3:47 - What is Databricks and its Architecture 5:33 - Medallion Architecture Explained 10:47 - Setting up Databricks in Azure 19:04 - Access Control and Security 29:04 - Connecting Databricks to Storage 36:07 - Introduction to Databricks Notebooks 44:24 - Setting up Compute Resources 45:54 - Configuring Compute Resources 48:02 - Installing Required Libraries 50:34 - Setting Up Notebooks 53:05 - Bronze Notebook: Fetching Raw Data 1:00:00 - Verifying Data in Bronze Layer 1:02:14 - Silver Notebook: Data Transformation 1:10:09 - Gold Notebook: Advanced Transformations 1:18:01 - Writing to Gold Layer 1:23:19 - Creating a Workflow in Databricks 1:27:25 - Configuring Task Dependencies 1:30:34 - Testing the Workflow 1:33:27 - Conclusion and Next Steps