Modern Spark DataFrame & Dataset | Apache Spark 2.0 Tutorial

Adam Breindel, lead Spark instructor at NewCircle, talks about which APIs to use for modern Spark with a series of brief technical explanations and demos that highlight best practices, latest APIs, and new features. (Topics Indexed Below) We'll look at how Dataset and DataFrame behave in Spark 2.0, Whole-Stage Code Generation, and go through a simple example of Spark 2.0 Structured Streaming (Streaming with DataFrames) that you can run in your own free instance of Databricks. 00:00:40 - Intro: What is "Modern Spark" 00:01:26 - DataFrame 00:05:07 - Why not use RDD? 00:09:15 - Intro to DataFrame and Dataset 00:10:13 - DataFrame versus Dataset 00:14:42 - Dataset Queries and Dataset with Scala classes 00:19:07 - Spark Query Optimizer 00:23:26 - Whole-Stage Codegen 00:27:21 - Hive integration 00:29:28 - Wrapping Up DataFrame/Dataset Benefits 00:30:54 - One More Thing - Structured Streaming 00:36:47 - Conclusion Try the Examples: + Databricks Community Edition: https://databricks.com/try + Get this Notebook: https://bit.ly/get-notebook ---------------------------------------------------------------------------------------------- SPARK 2.0 TRAINING | NewCircle | Onsite & Public Classes ---------------------------------------------------------------------------------------------- + Programming for Spark 2.0 (3 days): http://bit.ly/spark-prog-newcircle + Spark 2.0 for Machine Learning & Data Science (3 days): http://bit.ly/spark-ml-newcircle