Azure End-To-End Data Engineering Project for Beginners (FREE Account) | API Tutorial

💻 Beta test my AI App: https://forms.gle/tZ3mFxHEScktUwfq8 🤖 Early access to the Applied AI Community: https://kickofflabs.com/waitlist/f795d9da Get the resources (Github) 👉 https://www.lukejbyrne.com/c/fabric-vs-azure Join the Community 👉 https://www.youtube.com/channel/UCoaAe6ighxrzJuHeFPAlCeA/join This project addresses a critical business need by building a comprehensive data pipeline on Azure. The goal is to extract seismic data from an API endpoint via HTTP GET request, transform it in the cloud, and generate actionable insights through a Power BI dashboard. **Tech Stack:** HTTP, Python, PySpark, API, Data Factory, Data Lake Gen 2, Databricks, Synapse Analytics, Power BI, Entra ID (Active Directory), Key Vault # Watch Next https://youtu.be/P7EqW6_7wKs https://youtu.be/mbyZaxG3QZo https://youtu.be/lyp8rlpJc3k https://youtu.be/jofa6Yxd0pc # Courses & Certificates 👇 Check them out for free 👇 ## Data Engineer 🌍 Understanding Data Engineering 👉 https://datacamp.pxf.io/JKq0ZE 🏅 Database Design 👉 https://datacamp.pxf.io/Xm1AEg 📊 Data Warehousing Concepts 👉 https://datacamp.pxf.io/WyYaPX ⚡ Introduction to dbt 👉 https://datacamp.pxf.io/09Y7BJ 🕹️ Introduction to Apache Airflow in Python 👉 https://datacamp.pxf.io/nXn3eX 📈 Introduction to NoSQL 👉 https://datacamp.pxf.io/6yG9RG ## Cloud Engineer 🌤️ Understanding Cloud Computing 👉 https://datacamp.pxf.io/LKDqYY 1️⃣ Understanding Microsoft Azure 👉 https://datacamp.pxf.io/mOn3k1 2️⃣ AWS Concepts 👉 https://datacamp.pxf.io/7abrBQ 📊 Microsoft Azure Architecture and Services 👉 https://datacamp.pxf.io/MAQxYo ⚙️ Microsoft Azure Management and Governance 👉 https://datacamp.pxf.io/YRdGkj 🧩 AWS Cloud Technology and Services Concepts 👉 https://datacamp.pxf.io/4GdDBo # Social Media 📧 Newsletter: https://lukejbyrne.com/subscribe 🌄 Instagram: https://www.instagram.com/dataluke_ ⏰ TikTok: https://www.tiktok.com/@dataluke 📚 Linkedin: https://linkedin.com/in/lukejbyrne 🛍️ Amazon storefront: https://www.amazon.com/shop/lukejbyrne 🏆 1:1 Coaching: https://calendly.com/lukejbyrne/30min Some links included are affiliate links, which help me keep this channel going. Thanks for the support. Business inquiries: [email protected] How to set up PowerBI without work email: https://www.youtube.com/watch?v=9RB5xic9BiY Pathfinder's original vid: https://youtu.be/Av44Nrhl05s?si=y_CG7mLKh4PuBbe4 Overview --- 0:00 – Introduction & What You’ll Learn 0:22 – Main Azure Services Explained 0:54 – Getting Started & GitHub Repo 1:33 – Earthquake Data Pipeline Use Case 2:19 – Medallion Architecture Overview 2:45 – Synapse, Data Factory, & Power BI Databricks --- 3:33 – Prerequisites & Azure Free Account 5:16 – Creating the Azure Databricks Workspace 7:33 – Creating the ADLS Storage Account 10:03 – Deeper Dive Into Databricks Deployment 1:00:39 – Library Management & Reverse Geocoder 1:06:00 – Classifying Earthquake Significance Data Factory --- 1:09:05 – Data Factory Setup & Linking 1:24:26 – Passing Parameters & Orchestrating Notebooks Synapse --- 1:37:19 – Loading Data into Synapse Analytics 1:40:02 – Querying Parquet Data with OPENROWSET Power BI --- 1:44:42 – Power BI Overview & Next Steps