The Long-Horizon Holdout Protocol
by Archie Abrams • VP of Product and Head of Growth at Shopify
Leads an org of over 600 people across product, design, engineering, and growth. Previously at Udemy.
🎙️ Episode Context
Archie Abrams discusses Shopify's contrarian approach to growth and product development, including their refusal to use metrics for core product decisions, their 100-year planning horizon, and why they optimize for churn to maximize entrepreneurship.
Problem It Solves
Prevents the 'pull-forward' illusion where growth teams claim credit for revenue that would have happened anyway, and identifies features that degrade value over time despite initial excitement.
Framework Overview
A measurement framework that challenges short-term growth wins by maintaining long-term control groups (holdouts) to verify if immediate lifts translate to sustainable value (GMV) over 1-3 years.
📅 Framework Timeline
Institute a holdout group that does n...
Automate 'pings' to experimenters 6-1...
Prioritize Cohort GMV (Gross Merchand...
Accept that 30-40% of short-term 'win...
When to Use
When running experiments on platforms where user value compounds over a long period (e.g., SaaS, eCommerce, FinTech) rather than transactional loops.
Common Mistakes
Declaring victory after 2 weeks of statistical significance; assuming neutral short-term results mean the feature is a failure (it might compound later).
Real World Example
Shopify tested payment failure notifications. Short-term result: Huge lift in recovered payments. Long-term result (12 mo): Zero incremental GMV because those users weren't committed entrepreneurs and would have churned anyway.
If you can't measure it longer term, you'll get better about identifying what things are that are really impactful... look at the downstream metrics for a year, two years on that experiment.
— Archie Abrams