Answer.ai & AI Magic with Jeremy Howard

Jeremy Howard and Eric Ries founded Answer.AI to do exactly one thing: “Practical AI R&D”. One of their first releases was a system based on FSDP + QLoRA that let anyone train a 70B model on two NVIDIA 4090s. Since then, they have come out with a long list of super useful projects with a very small team. In today's episode we talked through the origin of Answer, some of their recent work, and their upcoming project "Magic AI". Full show notes: https://www.latent.space/p/answerai 00:00:00 Introduction 00:01:07 Continous Pre-Training is Here 00:04:48 Schedule-Free Optimizers and Learning Rate Schedules 00:06:08 Governance and Structural Issues within OpenAI and Other AI Labs 00:13:32 How Answer.ai works 00:27:04 How to Recruit Productive Researchers 00:32:34 Building a new BERT 00:37:10 FSDP, QLoRA, and QDoRA: Innovations in Fine-Tuning Large Models 00:43:42 Research and Development on Model Inference Optimization 00:47:48 FastHTML for Web Application Development 01:01:16 AI Magic & Dialogue Engineering 01:04:11 AI wishlist & predictions