The Open-to-Axial Analysis Loop
by Hamel Husain & Shreya Shankar • Co-Founders of the 'Build Your Own Evals' Course at Consulting / UC Berkeley
Hamel is a machine learning engineer with experience at GitHub and Airbnb, now a leading AI consultant. Shreya is a computer scientist and researcher at UC Berkeley, specializing in ML operationalization. Together, they run the top-rated course on Maven about building AI evaluations.
🎙️ Episode Context
This episode demystifies 'Evals' (evaluations) for AI products, arguing they are the highest ROI activity for AI teams. Hamel and Shreya demonstrate a practical workflow starting from manual error analysis ('open coding') to building automated 'LLM-as-a-Judge' systems. They challenge the misconception that evals are just unit tests, framing them instead as a continuous data analysis process that replaces traditional PRDs for AI agents.
Problem It Solves
Overcoming the 'cold start' problem where teams don't know what to test or fix in their AI product because raw logs are overwhelming.
Framework Overview
A qualitative research method adapted for AI logs. Instead of guessing failure modes, builders manually review production traces ('Open Coding') to tag issues freely, then cluster these tags into broader categories ('Axial Coding') to identify high-leverage areas for improvement.
🧠 Framework Structure
Read traces manually until 'Theoretic...
Write 'Open Codes' (free-form notes) ...
Use an LLM to synthesize Open Codes i...
When to Use
At the beginning of product development or whenever user feedback indicates quality issues but metrics are silent.
Common Mistakes
Skipping manual review to automate immediately, or using vague tags like 'janky' without specific context.
Real World Example
Analyzing a real estate agent bot (Nurture Boss) and discovering it was hallucinating 'virtual tours' and failing to hand off to humans during sensitive requests.
To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in.
— Hamel Husain & Shreya Shankar