The OEC Framework (Overall Evaluation Criterion)
by Ronny Kohavi • Author, Instructor, Former VP at Airbnb/Microsoft/Amazon at Independent / Maven Course Instructor
Ronny Kohavi is widely considered the 'godfather' of A/B testing and online experimentation. He previously led experimentation teams at Airbnb (VP), Microsoft (Corporate VP of Analysis & Experimentation), and Amazon, and co-authored the definitive book 'Trustworthy Online Controlled Experiments'.
🎙️ Episode Context
Ronny Kohavi dives deep into the science and culture of A/B testing, explaining why 80-90% of experiments fail and how to build a trustworthy experimentation platform. He discusses critical concepts like the Overall Evaluation Criterion (OEC), Twyman's Law, and the statistical pitfalls that mislead product teams.
Problem It Solves
Prevents teams from optimizing for short-term vanity metrics (like revenue or clicks) that degrade the user experience and hurt long-term retention.
Framework Overview
The OEC is a quantitative measure of the experiment's objective. It is a single metric (or a function of metrics) that aligns with the company's strategic goals and is causally predictive of long-term customer lifetime value.
🧠 Framework Structure
Align with Lifetime Value (LTV): Iden...
Constraint Optimization: Maximize the...
Countervailing Metrics: Always pair a...
When to Use
When designing any experiment, especially those involving monetization or engagement where negative side effects are possible.
Common Mistakes
Choosing 'Revenue' as the sole OEC without constraints, leading to a spammy user experience.
Real World Example
At Bing, they didn't just optimize for ad clicks; they optimized for revenue per search constrained by the number of pixels ads took up, ensuring the organic results weren't pushed too far down.
It's very easy to increase revenue by doing theatrics... but it hurts the user experience. You have to define the OEC such that it is causally predictive of the lifetime value of the user.
— Ronny Kohavi