The AEO Split-Test Protocol
by Ethan Smith • CEO at Graphite
An SEO veteran with 18 years of experience, Ethan specializes in SEO and programmatic SEO strategies. He is the CEO of Graphite, a leading agency helping companies like MasterClass and Webflow build scalable growth channels.
🎙️ Episode Context
Ethan Smith joins Lenny to decode the rapid shift from traditional SEO to Answer Engine Optimization (AEO). They explore how LLMs and RAG (Retrieval Augmented Generation) are changing search behavior, why 'share of voice' in citations matters more than ranking number one, and provide a tactical playbook for influencing AI answers through Reddit, YouTube, and help centers.
Problem It Solves
Eliminates the guesswork and misinformation surrounding AEO by scientifically validating which tactics actually influence LLM answers.
Framework Overview
A rigorous experimental design to measure 'Share of Voice' in LLM answers, distinguishing between actual algorithmic wins and random variance.
🧠 Framework Structure
Question Transformation: Convert high...
Share of Voice Tracking: Use an answe...
Control Group Isolation: Select 200 q...
Reproducibility Check: Intervention r...
When to Use
Before investing significant budget into a specific AEO strategy (like influencer marketing or massive content generation).
Common Mistakes
Failing to use a control group (LLM traffic is naturally rising, so 'up and to the right' might be a false positive) or testing on too few questions.
Real World Example
Ethan describes running this protocol where they tracked thousands of questions, intervened on half with tactics like Reddit comments, and measured the delta in 'Share of Voice' against the control group.
Most best practices, most blog posts are not correct. So how do you set up an experiment? ... Have a test group, have a control group. Intervene on the test group... see if the chart went up.
— Ethan Smith