Foundation Models | On the opportunities and risks of calling pre-trained models “Foundation Models”

Sound the opinionated video alarm! 🚨 We need to talk about “foundation models”: What does the term mean? Is ViT a foundation model? Do we really need AI to “understand”? And what’s the thing with out-of-domain generalization / distribution shift? 😎 Btw, 50,000 ViT models released with the "How to train your ViT" paper by Steiner et al. 2021. (see reference below 👇) Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 donor, Dres. Trost GbR ➡️ AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/ ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ Papers: 📜Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein et al. "On the Opportunities and Risks of Foundation Models." arXiv preprint arXiv:2108.07258 (2021). https://arxiv.org/abs/2108.07258 📜Steiner, Andreas, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. "How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers." arXiv preprint arXiv:2106.10270 (2021). https://arxiv.org/abs/2106.10270 📜 Zhai, Xiaohua, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. "Scaling vision transformers." arXiv preprint arXiv:2106.04560 (2021). https://arxiv.org/abs/2106.04560 Outline: 00:00 What is a foundation model? Is ViT one of them? 06:02 Foundation model paper highlights 07:02 Understanding 10:12 Data and distribution shift 14:00 Alignment and outro ---------------------------------- 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​ Thumbnail contains emojis designed by OpenMoji – the open-source emoji and icon project. License: CC BY-SA 4.0