Observability for Data Pipelines: Monitoring, Alerting & Tracing Lineage • Jiaqi Liu • GOTO 2020

This presentation was recorded at GOTO Chicago 2020. #GOTOcon #GOTOchgo http://gotochgo.com Jiaqi Liu - Leader in Bridging the Gap between Data Science and Engineering ABSTRACT Data-intensive applications, with many layers of transformations and movement from different data sources, can often be challenging to maintain even after they are initially built and validated. To truly expand and develop a code base, developers must be able to test confidently during the development process and monitor the production system. Monitoring and testing data pipelines or real-time streaming processes can be very different from monitoring web services. Jiaqi draws on her experience building and maintaining both batch and real-time stream data pipelines to discuss how to leverage monitoring tools like Prometheus and Grafana to define and visualize metrics, how and when to alert on common health indicators, and how to gain visibility in monitoring not just the system health but the health of the data. General concepts she touches on include observability of pipeline health, interpretability of data results and building features into data pipelines that makes monitoring and testing just a little [...] TIMECODES 00:00 Intro 00:56 Data Pipelines 03:20 What could go wrong? 05:16 Data pipeline concerns 06:14 Interpretability 07:42 Observability 08:44 Pipeline features 09:17 Immutable data 10:52 Data lineage 13:08 Dry run mode 15:21 Testing, monitoring & alerting Download slides and read the full abstract here: https://gotochgo.com/2020/sessions/1321/observability-for-data-pipelines-monitoring-alerting-and-tracing-lineage RECOMMENDED BOOKS Rishu Mehra • What is Data Observability • https://amzn.to/3zdkxFK Gerardus Blokdyk • Observability Services A Complete Guide • https://amzn.to/3zgA73A Bas P. Harenslak & Julian Rutger de Ruiter • Data Pipelines with Apache Airflow • https://amzn.to/3hrGH4a James Densmore • Data Pipelines Pocket Reference • https://amzn.to/3DU4qRU Barr Moses, Lior Gavish & Molly Vorwerck • Data Quality Fundamentals • https://amzn.to/3DSouUW https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #Prometheus #Grafana #Data #Programming #Backend #DevOps #Streams #Frontend CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1