Design Patterns for Running AIML and Bigdata Workloads on Kubernetes

Kubernetes is a defacto choice of resource manager for different types of requirements like running your microservices, CI/CD pipelines, Analytical applications, etc. Each of these requirements has specific challenges and as we address these challenges some new design patterns evolve which act as templates for many to use these learning and effectively run your workloads. Running AI/ML and Bigdata engines like Spark, Flink, Hive, Tez, Flume, etc on Kubernetes has presented multiple challenges with respect to handling container placement, Config management, confidential data management, enforcement of governance policies, Metadata management, Shuffle data management, Logging and monitoring, Autoscaling, Sharing data with other containers, Sharing libraries across different containers and many more. The Agenda of this talk is to discuss these challenges and how different design patterns have evolved to address them.