In this video we see how to perform data validation between datasets using PySpark and Databricks workflows.
You can find the Databricks Notebooks here:
https://github.com/apostolos1927/validation_framework
Follow me on social media:
LinkedIn: www.linkedin.com/in/apostolos-athanasiou-9a0baa119
GitHub: https://github.com/apostolos1927/
00:00 - Intro
01:34 - How to validate your data using PySpark