Distributionally Fair Rating System
by Ramesh Johari • Professor at Stanford University / Data Science Advisor at Stanford University (Advisor to Airbnb, Uber, Stitch Fix, etc.)
Ramesh Johari is a professor at Stanford University specializing in data science, game theory, and the design of online marketplaces. He has served as an advisor to major tech companies like Airbnb, Uber, and Upwork, helping them solve complex problems related to matching, pricing, and experimentation.
🎙️ Episode Context
Ramesh Johari dives deep into the science of building thriving marketplaces, arguing that marketplaces primarily sell the reduction of transaction friction rather than just goods. He explores the crucial distinction between predictive machine learning and causal decision-making, the nuances of designing fair rating systems, and why founders should focus on liquidity before platform dynamics.
Problem It Solves
Simple average ratings penalize new entrants (one bad review ruins them) and favor established players, reducing marketplace liquidity and competition.
Framework Overview
Design rating systems that account for the 'cold start' of reputation. Use priors to smooth out early variance and recognize that missing reviews ('sound of silence') contain predictive data.
🧠 Framework Structure
Avoid Raw Averages: Don't let a singl...
Leverage the Sound of Silence: Treat ...
Renorming Expectations: Ask specific ...
When to Use
Designing the trust/safety layer of a marketplace or gig economy platform.
Common Mistakes
Letting a new seller's business die because of one early negative review due to simple averaging logic.
Real World Example
Airbnb's 'double-blind' review system (reviews revealed simultaneously) which increased review rates by leveraging curiosity and reciprocity.
The biggest change... is that we get to see what happened with our matches.
— Ramesh Johari