Metadata-Driven Streaming Ingestion with Delta Live Tables, Azure Event Hubs & Azure Schema Registry

🎤 Vicky Avison, Principal Data Engineer, Plexure Talk Title: Metadata-Driven Streaming Ingestion with Delta Live Tables, Azure Event Hubs & Azure Schema Registry Talk Summary: At Plexure, we ingest hundreds of millions of customer activities and transactions into our data platform every day, fuelling our personalisation engine and providing insights into the effectiveness of marketing campaigns.We’re on a journey to transition from infrequent batch ingestion to near real-time streaming using Azure Event Hubs and Delta Live Tables. This transformation will allow us to react to customer behaviour as it happens, rather than hours or even days later.It also enables us to move faster in other ways. By leveraging a Schema Registry, we’ve created a metadata-driven framework that allows data producers to: - Evolve schemas with confidence, ensuring downstream processes continue running smoothly. - Seamlessly publish new datasets into the data platform without requiring Data Engineering assistance.In this talk, I'll talk about our journey and discuss how we’re implementing this with DLT meta-programming — including a live demo of the end-to-end process. Speaker Bio: Vicky is the Principal Data Engineer at Plexure, where she drives the technical vision and engineering standards for the data platform. She stumbled into the world of Big Data over 10 years ago and has been building and optimising data platforms and pipelines ever since.