End To End Big Data Engineering Project With Azure| Big Data Project From Scratch- Part 1

Hello All, Our first full fledged Big Data Bootcamp With Cloud Azure And AWS is live and has started from 21st December 2024. 𝐏𝐥𝐞𝐚𝐬𝐞 𝐟𝐢𝐧𝐝 𝐭𝐡𝐞 𝐜𝐨𝐮𝐫𝐬𝐞 𝐝𝐞𝐭𝐚𝐢𝐥𝐬 𝐛𝐞𝐥𝐨𝐰 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐖𝐢𝐭𝐡 𝐀𝐳𝐮𝐫𝐞 𝐀𝐧𝐝 𝐀𝐖𝐒 𝐂𝐥𝐨𝐮𝐝 𝐌𝐚𝐬𝐭𝐞𝐫𝐲 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 𝐌𝐞𝐧𝐭𝐨𝐫: Mayank Aggarwal 𝐒𝐭𝐚𝐫𝐭 𝐃𝐚𝐭𝐞: 𝐃𝐞𝐜𝐞𝐦𝐛𝐞𝐫 21𝐬𝐭 2024 𝐓𝐢𝐦𝐢𝐧𝐠: 8𝐚𝐦 𝐭𝐨 12𝐩𝐦 𝐈𝐒𝐓(𝐒𝐚𝐭𝐮𝐫𝐝𝐚𝐲 𝐀𝐧𝐝 𝐒𝐮𝐧𝐝𝐚𝐲) 𝐏𝐫𝐨𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐋𝐞𝐯𝐞𝐥 - 𝐏𝐫𝐨𝐟𝐞𝐬𝐬𝐢𝐨𝐧𝐚𝐥𝐬 (2+𝐲𝐞𝐚𝐫𝐬 𝐞𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞) Please find the course details below, https://bit.ly/41d7c1Q Code for this project: https://github.com/mayank953/BigDataProjects/tree/main/Project-Brazillian%20Ecommerce Timestamp 00:00:00 Project Introduction 00:50:00 Getting Started 02:34:00 Pre-requisite 08:34:00 Project Architecture 00:20:00 Azure Free Account 00:27:00 Azure Cloud 00:34:37 Olist Dataset 00:42:00 SQL DB & Data Ingestion 01:06:30 Role of Data Engineer 01:11:24 Resource & Resource Group 01:12:24 Azure Data Factory 01:31:00 ADLS storage account 01:41:00 Meddallion Architecture 01:46:23 Ingestion with ADF 02:00:46 Real Time Ingestion with ADF 02:26:49 Parametrized Ingestion ADF 02:42:00 Azure Databricks Creation 02:43:21 Databricks Overview 02:51:36 Databricks UI Overview 02:58:12 Compute and Notebook 03:00:00 MongoDB & Ingestion 03:08:10 Databricks Workflow 03:12:12 Datbricks to ADLS connection 03:25:12 Accessing data from Lake 03:30:02 Outro & Next Part