Modified Newton method | Exact Line Search | Theory and Python Code | Optimization Algorithms #4

☕️ Buy me a coffee: https://paypal.me/donationlink240 🙏🏻 Support me on Patreon: https://www.patreon.com/c/ahmadbazzi In this one, I will show you what the modified newton algorithm is and how to use it with the exact line search method. We will approach both methods from intuitive and animated perspectives. The difference between Damped and its modified newton method is that the Hessian may run into singularities at some iterations, and so we apply diagonal loading, or Tikhonov regularization at each iteration. As a reminder, Damped newton, just like newton’s method, makes a local quadratic approximation of the function based on information from the current point, and then jumps to the minimum of that approximation. Just imagine fitting a little quadratic surface in higher dimensions to your surface at the current point, and then going to the minimum of the approximation to find the next point. Finding the direction towards the minimum of the quadratic approximation is what you are doing. As a matter of fact, this animation shows you why in certain cases, Newton's method can converge to a saddle or a maximum. If the eigenvalues of the Hessian are non positive - in those cases the local quadratic approximation is an upside down paraboloid. ⏲Outline⏲ 00:00 Introduction 00:55 Modified Newton Method 03:41 Exact line search 04:55 Python Implementation 18:46 Animation Module 34:14 Animating Iterations 37:02 Outro 📚Related Courses: - 📚 Convex Optimization Extended Course https://www.youtube.com/playlist?list=PL-DDW8QIRjNOVxrU2efygBw0xADVOgpmw - 📚 Python Programming Extended Course https://www.youtube.com/playlist?list=PL-DDW8QIRjNOXfBOPL67RCjgTELw4Y58F - 📚 Convex Optimization Applications Extended Course https://www.youtube.com/watch?v=38u85fHxU-M&list=PL-DDW8QIRjNOfSM_LTe1o7-lg1PIuw-0R&ab_channel=AhmadBazzi - 📚 Linear Algebra Extended Course https://www.youtube.com/playlist?list=PL-DDW8QIRjNOv5V6wqyCn781CY8_znkKh - 📚 Python projects course https://www.youtube.com/playlist?list=PL-DDW8QIRjNM-q7Tu6ce6dmeTm0fS818H 🔴 Subscribe for more videos on CUDA programming 👍 Smash that like button, in case you find this tutorial useful. 👁‍🗨 Speak up and comment, I am all ears. 💰 If you are able to, donate to help the channel Patreon - https://www.patreon.com/ahmadbazzi #python #optimization #algorithm