Problem-First Workflow Analysis
by Keith Coleman & Jay Baxter • Product Lead & Founding ML Engineer at Community Notes (X)
Keith Coleman and Jay Baxter lead the Community Notes team at X (formerly Twitter). Keith, a former product leader, and Jay, a machine learning engineer, developed the open-source, crowd-sourced context system that uses bridging algorithms to combat misinformation.
🎙️ Episode Context
Keith Coleman and Jay Baxter discuss the origins, principles, and engineering behind Community Notes. They reveal how a small, 'thermal' team used dynamic goal-setting and a unique bridging-based algorithm to build a decentralized fact-checking system that scales effectively where traditional methods failed. The conversation covers the transition from Birdwatch, the impact of open-sourcing the algorithm, and the operational philosophy of 'Continuous Calibration' that allows them to move rapidly.
Problem It Solves
Avoids building expensive, unscalable solutions based on assumptions or industry status quo.
Framework Overview
A rigorous development lifecycle that starts with a fundamental problem definition and demands distinct proof-of-concept validation at every stage before scaling. It avoids adopting industry norms (like human fact-checkers) unless they survive the problem analysis.
⚡ Step-by-Step Framework
Problem Obsession: Focus entirely on solving the core issue (e.g., info quality) rather than business metrics.
Stepwise Proof: The product must prove its value hypothesis at each stage (Mockup -> Pilot -> Scale).
First-Principles Design: Reject existing solutions (Trust & Safety teams) if they don't solve the scale/trust problem.
Radical Transparency: Open-sourcing code and data to build trust rather than relying on brand authority.
Problem Obsession: Focus entirely on solving the core issue (e.g., info quality) rather than business metrics.
Stepwise Proof: The product must prove its value hypothesis at each stage (Mockup -> Pilot -> Scale).
First-Principles Design: Reject existing solutions (Trust & Safety teams) if they don't solve the scale/trust problem.
Radical Transparency: Open-sourcing code and data to build trust rather than relying on brand authority.
When to Use
Developing zero-to-one products in high-stakes environments where trust is low and scale is high.
Common Mistakes
Scaling before proving the core mechanic; relying on 'black box' logic that users cannot verify.
Real World Example
The team testing Community Notes first as static mockups to prove demand, then as an MTurk task to prove capability, before building the live pilot.
We were very disciplined... about having the product prove itself at every given point.
— Keith Coleman & Jay Baxter