Twyman's Law & Data Trust Validation
by Ronny Kohavi • Author, Instructor, Former VP at Airbnb/Microsoft/Amazon at Independent / Maven Course Instructor
Ronny Kohavi is widely considered the 'godfather' of A/B testing and online experimentation. He previously led experimentation teams at Airbnb (VP), Microsoft (Corporate VP of Analysis & Experimentation), and Amazon, and co-authored the definitive book 'Trustworthy Online Controlled Experiments'.
🎙️ Episode Context
Ronny Kohavi dives deep into the science and culture of A/B testing, explaining why 80-90% of experiments fail and how to build a trustworthy experimentation platform. He discusses critical concepts like the Overall Evaluation Criterion (OEC), Twyman's Law, and the statistical pitfalls that mislead product teams.
Problem It Solves
Identifies invalid experiments that look successful due to instrumentation errors, preventing teams from celebrating false wins.
Framework Overview
Twyman's Law states that 'Any figure that looks interesting or different is usually wrong.' This framework requires rigorous validation of outliers before accepting them as true results.
🧠 Framework Structure
Sample Ratio Mismatch (SRM) Test: Che...
Hold the Celebration: If a result exc...
Segment Analysis: Break down the 'win...
Guardrail Metrics: Monitor technical ...
When to Use
Whenever an experiment returns a statistically significant result, especially one that is surprisingly positive.
Common Mistakes
Accepting a 'Sample Ratio Mismatch' result because the lift is high, ignoring that the population bias invalidates the math.