Thinking in Partitions (A Dagster Deep Dive)

By embracing partitioning, you can speed up execution and greatly reduce the cost of running pipelines. Gain control over backfills and observe your data assets at a higher level of granularity. You can catch all Dagster Deep Dives in the series here: https://dagster.io/events/dagster-deep-dives-feb-2024 You will find the Dagster repo for these sessions here: https://bit.ly/DagDemo Be sure to bookmark (star) the Dagster repo for future reference: https://github.com/dagster-io/dagster 00:00 Intro 01:22 Why partition your pipeline? 05:06 An intro to Dagster's partitions API 06:37 Partitioning by category 11:24 Partition an asset by time 15:23 Recap of partitioning and partitioning patterns. 22:00 wrap up and other resources 22:40 Q&A