💡

InsightHunt

Hunt the Insights

K

Kevin Weil

Episode #172

Chief Product Officer

OpenAI

🎯Product Strategy⚡Execution🔍User Research

📝Full Transcript

17,322 words
Kevin Weil (00:00:00): The AI models that you're using today is the worst AI model you will ever use for the rest of your life, and when you actually get that in your head, it's kind of wild. Everywhere I've ever worked before this, you kind of know what technology you're building on, but that's not true at all with AI. Every two months, computers can do something they've never been able to do before and you need to completely think differently about what you're doing. Lenny Rachitsky (00:00:21): You're chief product officer of maybe the most important company in the world right now. I want to chat about what it's just like to be inside the center of the storm. Kevin Weil (00:00:29): Our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are. And we say this to developers too. If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right. Give it another couple months and the models are going to be great, and suddenly the product that you have that just barely worked is really going to sing. Lenny Rachitsky (00:00:51): Famously, you led this project at Facebook called Libra. Kevin Weil (00:00:56): Libra is probably the biggest disappointment of my career. It fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product. We tried to launch a new blockchain. It was a basket of currencies originally. It was integration into WhatsApp and Messenger. I would be able to send you 50 cents in WhatsApp for free. It should exist. To be honest, the current administration is super friendly to crypto. Facebook's reputation is in a very different place. Maybe they should go build it now. Lenny Rachitsky (00:01:27): Today my guest is Kevin Weil. Kevin is chief product officer at OpenAI, which is may...

💡 Key Takeaways

  • 1The AI models you use today are the worst you will ever use; build for where the puck is going, not current limitations.
  • 2Writing 'evals' (evaluations) is becoming a core competency for Product Managers to measure model performance on specific use cases.
  • 3Adopt 'Iterative Deployment': ship early to co-evolve with society rather than waiting for perfection.
  • 4Treat AI models like humans: use 'chain of thought' for reasoning and 'ensembles' of models like a team of experts.
  • 5PMs need to be 'High Agency' and comfortable with extreme ambiguity; OpenAI keeps the PM-to-Engineer ratio low to prevent micromanagement.
  • 6Future product teams will embed researchers and ML engineers to handle fine-tuning for specific industry data.

📚Methodologies (3)

🎯 Product Strategy

A strategic mindset that assumes AI models will improve drastically every few months. Instead of building complex workarounds for current flaws, teams should build products that push the edge of current capabilities, knowing the next model update will likely solve the friction points.

Core Principles

  • 1.Build on the edge: If your product barely works today, it will 'sing' with the next model update.
  • 2.Ignore temporary flaws: Don't spend excessive resources fixing errors that model scaling will naturally solve.
  • 3.Iterative Deployment: Ship continuously to learn how users interact with the model as it improves.

"If the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right."

#maximalism#strategy#product
View Deep Dive →
⚡ Execution

Product Managers must define 'hero use cases' and translate them into specific evaluations (evals)—essentially quizzes for the model. Development becomes a process of hill-climbing on these eval scores, often using fine-tuning to improve performance on specific tasks.

Core Principles

  • 1.Define Hero Use Cases: Identify the specific complex queries or tasks the product must solve.
  • 2.Create Custom Evals: Build a dataset of questions and 'perfect' answers to grade the model.
  • 3.Fine-tune & Hill Climb: Use the data to teach the model and measure progress against the evals continuously.

"Writing evals is quickly becoming a core skill for product builders."

#evals-driven#development#cycle
View Deep Dive →
🔍 User Research

Design interfaces and workflows by asking 'How would a human handle this?' This informs decisions on latency (showing thinking process vs. silence), error handling, and using chat as a universal interface because it matches human communication flexibility.

Core Principles

  • 1.Show the Work: Just as a human says 'Let me think about that,' the UI should show the model's reasoning steps (summarized) to manage wait times.
  • 2.Ensemble Intelligence: Treat different models like a team of experts (one for coding, one for writing) and use an orchestrator to combine their outputs.
  • 3.Chat as Universal Interface: Chat handles the nuances of intelligence better than rigid buttons.
  • +1 more...

"You can often reason about it the way you would reason about another human and it works."

#human-analog#interaction#design
View Deep Dive →
← Browse All Episodes