Josh Tobin: LLMOps: Test-Driven Development for Large Language Model Applications

ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Large language models are a powerful primitive for building applications quickly and easily. However, when it comes to robustness, reliability, and production readiness, they leave something to be desired. If you've built applications with LLMs, you may have wondered, "isn't it a bit generous to call this prompt engineering?", "how do I know if this thing is actually working", or "is it even possible to test these things"? In this talk, we will present a more principled way to develop LLM applications using an approach that is analogous to test-driven development. We'll also show you how to get started with this approach in minutes using Gantry.