by Shaun Clowes
When analyzing a metric change or experiment result, never look at the data point in isolation. You must analyze the context surrounding the data—where users came from, where they went, and the higher-level business impact.
Core Principles
- 1.Check Upstream: Look at what happened before the event. Is there selection bias? Does this only apply to 2% of traffic?
- 2.Check Downstream: Look at what happens after. Do these users retain in week 3, or do they churn immediately after the 'success' event?
- 3.Click Up (Macro View): Look at the business level. Did this increase users but lower the Average Sales Price (ASP) or revenue quality?
"If you look at data as a way of giving you the answer, you're always wrong... Data is more like a compass than a GPS."