Spark Structured Streaming Integration With Event Hubs

This tutorial explores Spark Structured Streaming's Integration with Azure Event Hubs. I explain key terms and concepts of Event Hubs and demonstrate that event payload data from Event Hubs can be ingested, transformed, and written into delta tables. Furthermore, I explain how to write to Event Hubs from Spark Structured Streaming. Chapters: 00:00- Introduction 02:12- Overview of Azure Event Hubs 07:39- Course setup 14:16- Reading from Event Hubs 18:08- Reading a subset of events from Event Hubs 19:49- Converting event payload to readable format 21:31- Parsing event payload data and writing to Delta table 25:46- Applying time-based aggregations on streams 27:26- Writing streams to Event Hubs Please subscribe: https://www.youtube.com/channel/UC8d958MxE2t1dr27QNqoOhA Download demo/exercise notebooks from here: https://github.com/fazizov/youtube/tree/44a5ae06febe9c473f716e2ca5dbfc8c7cf4b3bb/Data%20engineering%20with%20Databricks/Spark%20Streaming%20Event%20Hubs