AWS re:Invent 2024 - Efficient incremental processing with Apache Iceberg at Netflix (NFX303)

Incremental processing substantially reduces compute costs and execution time, leading to fewer failures and less manual intervention. However, enabling it on large-scale data workflows presents significant challenges around scalability, ease of adoption, and user experience. In this session, learn how Apache Iceberg and Netflix Maestro are being leveraged to build an incremental processing solution (IPS) that processes new or changed data, reducing costs and time while ensuring data accuracy and freshness. Combining Iceberg’s metadata capabilities with Maestro’s workflow orchestration allows for the efficient management of late-arriving data and backfills in various different scenarios beyond append-only mode. Learn more: AWS re:Invent: https://go.aws/reinvent. More AWS events: https://go.aws/3kss9CP Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4 About AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster. #AWSreInvent #AWSreInvent2024