Practical AB testing advices from Sr Staff data scientist at Meta | with John Meakin, Part 1

Insights and Mistakes in Running Effective Experimentations The script delves into common mistakes and misconceptions associated with running experimentations, primarily in the tech industry. It highlights the importance of experimental setup and design over data interpretation, addressing common pitfalls such as improper experimental implementation and the misinterpretation of statistical significance. The conversation also touches upon the backgrounds of individuals in the field, emphasizing learning through experience over formal education. Further, it explores nuanced issues like sample ratio mismatches, the impact of company culture on experimentation practices, and the necessity of a principled approach to experimentation. The underlying theme suggests a balance between statistical rigor and practical business value, advocating for a disciplined yet flexible approach to experimenting in a corporate setting. 00:00 Exploring Common Mistakes in Experimentation 01:19 Personal Journeys into Experimentation 02:39 Advice for New Data Scientists on Experimentation 04:02 Misconceptions and Overemphasized Issues in Experimentation 07:44 The Importance of Experiment Design Over Data Interpretation 14:34 Understanding and Misunderstanding Statistical Concepts 19:09 Balancing Business Needs with Statistical Rigor