LLMOps (LLM Bootcamp)

New course announcement ✨ We're teaching an in-person LLM bootcamp in the SF Bay Area on November 14, 2023. Come join us if you want to see the most up-to-date materials building LLM-powered products and learn in a hands-on environment. https://www.scale.bythebay.io/llm-workshop Hope to see some of you there! --------------------------------------------------------------------------------------------- In this video, Josh gives a tour of the emerging discipline of LLMOps: principles and practices for continuous improvement of large language model-powered applications. - Comparing and evaluating open source and proprietary models - Workflows and tools for iteration and prompt management - Principles for applying test-driven development to LLMs Download slides and view lecture notes: https://fullstackdeeplearning.com/llm-bootcamp/spring-2023/llmops/ Intro and outro music made with Riffusion: https://github.com/riffusion/riffusion Watch the rest of the LLM Bootcamp videos here: https://www.youtube.com/playlist?list=PL1T8fO7ArWleyIqOy37OVXsP4hFXymdOZ 00:00 Why LLMOps? 01:55 Choosing your base LLM 04:20 Proprietary LLMs 09:15 Open-source LLMs 14:45 Iteration and prompt management 22:35 Testing LLMs: Why and why is it hard? 28:15 Testing LLMs: What works? 35:05 Evaluation metrics for LLMs 37:40 Deployment and monitoring 44:35 Test-driven development for LLMs