by Archie Abrams
A measurement framework that challenges short-term growth wins by maintaining long-term control groups (holdouts) to verify if immediate lifts translate to sustainable value (GMV) over 1-3 years.
Core Principles
- 1.Institute a holdout group that does not receive the feature for a significant duration (12-18 months).
- 2.Automate 'pings' to experimenters 6-12 months post-launch to review long-term data.
- 3.Prioritize Cohort GMV (Gross Merchandise Value) over short-term conversion rates.
- +1 more...
"If you can't measure it longer term, you'll get better about identifying what things are that are really impactful... look at the downstream metrics for a year, two years on that experiment."