Uncertainty Quantification (4A): Implementing Split Conformal - Relation for Prediction Intervals

#uncertainty #datascience #statistics #predictioninterval #conformalprediction #conformal In the last two episodes, we explored the concept of measuring the non-conformity of a point with a bag of points using methods like absolute errors and the conformity ladder. In this video, we delve deeper into split conformal methods, aiming to demonstrate how to add uncertainty quantification to any model with just a few lines of code. Before we proceed, I highly recommend watching the previous episodes for a better understanding. The core of this episode revolves around an essential statement we discussed before: the 80% prediction interval for a test or validation point with an unknown label consists of plausible values that place it at the top 80% of the conformity ladder. We explore different ways to construct this interval and present a more intelligent approach that doesn't require trying all plausible values. We introduce intrinsic confidence levels and non-conformity quantiles, which offer a systematic way to determine the widths of prediction intervals. By understanding their relationship, we derive an easy-to-implement relation for prediction intervals using split conformal methods. Join us in part two of this episode, where we demonstrate how simple it is to implement this relation. Thanks for watching, and stay tuned for more valuable insights! 00:00 - Recap of Previous Episodes 00:53 - Final Goal of this Episode and One Following 02:20 - How to Construct Prediction Intervals 02:30 - Brute Force Implementation 03:16 - Intelligent Implementation 04:42 - How to Predict Upper and Lower Bounds? 08:42 - Role of Exchangeability 10:06 - Intrinsic vs. Arbitrary Confidence Levels 11:42 - Calibration Non-Conformities are Valid Quantiles 13:40 - Relation between Non-Conformity Quantiles and Intrinsic Confidence Levels 14:31 - Deriving the Interval Relation for Intrinsic Confidence Levels 17:45 - Deriving the Interval Relation for Arbitrary Confidence Levels 25:14 - Summary Additional Resources For a nice discussion on the upper and lower bounds of coverage and how to check for validity, see Angelopoulos, Anastasios N., and Stephen Bates. "A gentle introduction to conformal prediction and distribution-free uncertainty quantification." arXiv preprint arXiv:2107.07511 (2021). For more conformal prediction resources, see "Awesome Conformal Prediction" repository https://github.com/valeman/awesome-conformal-prediction by Valery Manokhin.