Variational Inference | Evidence Lower Bound (ELBO) | Intuition & Visualization

In real-world applications, the posterior over the latent variables Z given some data D is usually intractable. But we can use a surrogate that is close to it. How do we optimize for it? Here are the notes: https://raw.githubusercontent.com/Ceyron/machine-learning-and-simulation/main/english/probabilistic_machine_learning/vi_and_elbo.pdf Here is the link to the interactive elbo plot: https://share.streamlit.io/ceyron/machine-learning-and-simulation/main/english/probabilistic_machine_learning/elbo_interactive_plot.py If you want to run the Python script yourself which requires you to have streamlit, plotly and TensorFlow Probability installed, you can find it here: https://github.com/Ceyron/machine-learning-and-simulation/blob/main/english/probabilistic_machine_learning/elbo_interactive_plot.py ------- 📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): https://github.com/Ceyron/machine-learning-and-simulation 📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: https://www.linkedin.com/in/felix-koehler and https://twitter.com/felix_m_koehler 💸 : If you want to support my work on the channel, you can become a Patreon here: https://www.patreon.com/MLsim ------- Timestamps: 00:00 Introduction 00:54 Problem of intractable posteriors 02:10 Fixing the observables X 02:29 The "inference" in variational inference 03:29 The problem of the marginal 05:06 Remedy: A Surrogate Posterior 06:11 The "variational" in variational inference 06:38 Optimizing the surrogate 08:47 Recap: The KL divergence 09:42 We still don't know the posterior 10:35 Deriving the ELBO 15:17 Discussing the ELBO 17:59 Defining the ELBO explicitly 18:24 When the ELBO equals the evidence 18:56 Equivalent optimization problems 20:38 Rearranging for the ELBO 21:08 Plot: Intro 22:32 Plot: Adjusting the Surrogate 24:02 Summary & Outro