PySpark Meets Velox: Redefining Efficiency in ML Workloads - Konstantinos Karanasos, Meta

AI applications within Meta, including recommendation systems and GenAI, are pushing the limits of our batch compute engines. As demand has grown exponentially over the past few years, bespoke solutions have emerged, but they often present challenges in terms of maintenance, reusability, efficiency, and reliability. In this talk, we introduce PySpark/Velox, a new offering at Meta that combines Python’s ease-of-use, Spark’s scalability, and Velox’s efficiency. We will discuss the architecture, various use cases and results, and how it fits into our existing stack.