FNet: Mixing Tokens with Fourier Transforms – Paper Explained

Ms. Coffee Bean explains and visualizes the paper "FNet: Mixing Tokens with Fourier Transforms" and what it all has to do with 🍰. Get some coffee and join Ms. Coffee Bean for this video! ➡️ AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/ ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ Discussed paper: 📄 Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontanon, S. (2021). FNet: Mixing Tokens with Fourier Transforms. https://arxiv.org/pdf/2105.03824.pdf Referenced videos: 📺 Ms. Coffee Bean explains the Transformer: https://youtu.be/FWFA4DGuzSc 📺 ViLBERT (Vision and Language with Transformers): https://youtu.be/dd7nE4nbxN0 📺 ViT: An image is worth 16x16 words: https://youtu.be/DVoHvmww2lQ 📺 Multimodality: https://youtu.be/P23EWdiPWDw Outline: * 00:00 Why attention is "meh" * 02:46 How does FNet work? * 05:28 How does Fourier help? * 08:15 FNet performance Other references: 📄 Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., ... & Dosovitskiy, A. (2021). MLP-Mixer: An all-MLP Architecture for Vision. arXiv preprint arXiv:2105.01601. https://arxiv.org/abs/2105.01601 📄 Liu, Hanxiao; Dai, Zihang; So, David R.; Le, Quoc V. "Pay Attention to MLPs" (2021) https://arxiv.org/abs/2105.08050 Music 🎵 : Cuttin It Close by DJ Freedem -------------------------------------------------- 🔗 Links: YouTube: https://www.youtube.com/AICoffeeBreak Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​