Going beyond RAG: Extended Mind Transformers - Phoebe Klett

Retrieval Augmented Generation is such a hack. Why would an embedding of your prompt coincide with the documents needed to answer it? Meanwhile Transformers already have a key/query mechanism built in! In this talk, we'll introduce Extended Mind Transformers, a new flavor of transformer that allows the model to select and attend to the most relevant information at each generation step. We demonstrate EMT's state-of-the-art performance and discuss important design decisions for long context applications. Github: https://github.com/normal-computing/extended-mind-transformers HF: https://huggingface.co/collections/normalcomputing/extended-mind-transformers-6655e9ba5853d86b32793aaf Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025 About Phoebe ML Engineer with background in abstract mathematics. Building language models that natively reason.