Institutional Memory & The 80/20 Innovation Rule
by Ronny Kohavi • Author, Instructor, Former VP at Airbnb/Microsoft/Amazon at Independent / Maven Course Instructor
Ronny Kohavi is widely considered the 'godfather' of A/B testing and online experimentation. He previously led experimentation teams at Airbnb (VP), Microsoft (Corporate VP of Analysis & Experimentation), and Amazon, and co-authored the definitive book 'Trustworthy Online Controlled Experiments'.
🎙️ Episode Context
Ronny Kohavi dives deep into the science and culture of A/B testing, explaining why 80-90% of experiments fail and how to build a trustworthy experimentation platform. He discusses critical concepts like the Overall Evaluation Criterion (OEC), Twyman's Law, and the statistical pitfalls that mislead product teams.
Problem It Solves
Prevents repeating failed experiments (sunk costs) and balances incremental optimization with high-risk innovation.
Framework Overview
A cultural framework where teams accept a high failure rate (80%+) for bold ideas, document 'surprising' failures to learn, and allocate resources between optimization and moonshots.
🧠 Framework Structure
Accept High Failure Rates: In mature ...
Document Surprises: Create a searchab...
Portfolio Allocation: Allocate ~70-80...
Incremental Redesigns: Avoid massive ...
When to Use
When planning product roadmaps and conducting quarterly reviews of team performance.
Common Mistakes
Punishing teams for failed experiments, which leads to them only testing 'safe' trivial changes.
Real World Example
Microsoft tried integrating social (Facebook/Twitter) into Bing search for 1.5 years. It failed completely. Documenting this prevents them from trying the exact same losing strategy again without new evidence.