💡

InsightHunt

Hunt the Insights

R

Ronny Kohavi

Author, Instructor, Former VP at Airbnb/Microsoft/Amazon

Independent / Maven Course Instructor

📈 Growth & Metrics (1)⚡ Execution (1)👥 Team & Culture (1)

Key Takeaways

  • 1.Most experiments fail (60-90%), so you must optimize for experiment velocity and low marginal cost.
  • 2.Do not optimize solely for revenue; use an OEC (Overall Evaluation Criterion) that balances long-term user value.
  • 3.If a result looks too good to be true, it is likely a data error (Twyman's Law).
  • 4.You need tens of thousands of users to detect large effects, but ~200k+ for robust continuous optimization.
  • 5.Trust is the most important currency in an experimentation platform; use guardrail metrics like Sample Ratio Mismatch (SRM).

Methodologies(3)

📈 Growth & Metrics

The OEC is a quantitative measure of the experiment's objective. It is a single metric (or a function of metrics) that aligns with the company's strategic goals and is causally predictive of long-term customer lifetime value.

Core Principles

  • 1.Align with Lifetime Value (LTV): Identify short-term metrics that predict long-term success (e.g., successful sessions vs. just clicks).
  • 2.Constraint Optimization: Maximize the goal (e.g., revenue) subject to constraints (e.g., max ad pixels per page).
  • 3.Countervailing Metrics: Always pair a success metric with a 'drag' metric (e.g., email revenue vs. unsubscribe rate).

"It's very easy to increase revenue by doing theatrics... but it hurts the user experience. You have to define the OEC such that it is causally predictive of the lifetime value of the user."

#(overall#evaluation#criterion)
View Deep Dive →
⚡ Execution

Twyman's Law states that 'Any figure that looks interesting or different is usually wrong.' This framework requires rigorous validation of outliers before accepting them as true results.

Core Principles

  • 1.Sample Ratio Mismatch (SRM) Test: Check if the ratio of users in Control vs. Treatment matches the design (e.g., 50/50). If not, the experiment is invalid.
  • 2.Hold the Celebration: If a result exceeds normal variance (e.g., +10% lift when +1% is normal), assume it's a bug first.
  • 3.Segment Analysis: Break down the 'win' to see if it's driven by bots, specific browsers, or redirects.
  • +1 more...
#twyman's#trust#validation
View Deep Dive →
👥 Team & Culture

A cultural framework where teams accept a high failure rate (80%+) for bold ideas, document 'surprising' failures to learn, and allocate resources between optimization and moonshots.

Core Principles

  • 1.Accept High Failure Rates: In mature products, 80-90% of experiments will fail to move the metric. This is a feature, not a bug.
  • 2.Document Surprises: Create a searchable library of experiments. Focus on 'surprising' results (where actual differs from predicted), not just wins.
  • 3.Portfolio Allocation: Allocate ~70-80% of resources to incremental wins (low risk) and 20% to high-risk/high-reward bets.
  • +1 more...
#institutional#memory#80/20
View Deep Dive →