Spark Monitoring: Basics

In this video I cover how to use the default UI for monitoring Apache Spark. I use Azure Databricks to demonstrate, but most of the methods are the same in any Spark environment. I focus most on how I use it in my day to day work as a Data Engineer. Topics included: Spark UI, query plan basic, Ganglia Metrics, and driver logs. Monitoring Streams with Custom Streaming Query Listener: https://youtu.be/iqIdmCvSwwU?t=268 More from Dustin: Website: dustinvannoy.com Twitter: @dustinvannoy Github: https://github.com/datakickstart