AI Engineering with Chip Huyen

On today’s episode of The Pragmatic Engineer, I’m joined by Chip Huyen, a computer scientist, author of the freshly published O’Reilly book AI Engineering, and an expert in applied machine learning. Chip has worked as a researcher at Netflix, was a core developer at NVIDIA (building NeMo, NVIDIA’s GenAI framework), and co-founded Claypot AI. She also taught Machine Learning at Stanford University. In this conversation, we dive into the evolving field of AI Engineering and explore key insights from Chip’s book, including: • How AI Engineering differs from Machine Learning Engineering • Why fine-tuning is usually not a tactic you’ll want (or need) to use • The spectrum of solutions to customer support problems – some not even involving AI! • The challenges of LLM evals (evaluations) • Why project-based learning is valuable—but even better when paired with structured learning • Exciting potential use cases for AI in education and entertainment • And more! — Brought to by: • Swarmia — The engineering intelligence platform for modern software organizations https://www.swarmia.com/pragmatic/ • Graphite — The AI developer productivity platform https://gt.dev/pragmatic • Vanta — Automate compliance and simplify security with Vanta http://vanta.com/pragmatic — The Pragmatic Engineer deepdives relevant for this episode: • Applied AI Software Engineering: RAG https://newsletter.pragmaticengineer.com/p/rag • How do AI software engineering agents work? https://newsletter.pragmaticengineer.com/p/ai-coding-agents • AI Tooling for Software Engineers in 2024: Reality Check https://newsletter.pragmaticengineer.com/p/ai-tooling-2024 • IDEs with GenAI features that Software Engineers love https://newsletter.pragmaticengineer.com/p/ide-that-software-engineers-love — Where to find Chip Huyen: • X: https://x.com/chipro • LinkedIn: https://www.linkedin.com/in/chiphuyen/ • Website: https://huyenchip.com/ Where to find Gergely Orosz: • X: https://x.com/GergelyOrosz • LinkedIn: https://www.linkedin.com/in/gergelyorosz/ • Bluesky: https://bsky.app/profile/gergely.pragmaticengineer.com • Newsletter and blog: https://www.pragmaticengineer.com/ — In this episode, we cover: (00:00) Intro (01:31) A quick overview of AI Engineering (06:45) How Chip ensured her book stays current amidst the rapid advancements in AI (11:35) A definition of AI Engineering and how it differs from Machine Learning Engineering (18:15) Simple first steps in building AI applications (24:38) An explanation of BM25 (retrieval system) (25:28) The problems associated with fine-tuning (29:40) Simple customer support solutions for rolling out AI thoughtfully (35:29) Chip’s thoughts on staying focused on the problem (37:04) The challenge in evaluating AI systems (40:03) Use cases in evaluating AI (43:09) The importance of prioritizing users’ needs and experience (48:09) Common mistakes made with Gen AI (53:57) A case for systematic problem solving (54:57) Project-based learning vs. structured learning (1:00:07) Why AI is not the end of engineering (1:04:56) How AI is helping education and the future use cases we might see (1:08:58) Rapid fire round — See the transcript and other references from the episode at https://newsletter.pragmaticengineer.com/podcast — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].