Key Takeaways

  • Code generation tools have made writing software cheap, shifting the product manager's primary edge from feature coordination to editorial taste and noise filtering.
  • At Google, PMs working on Gemini no longer write traditional requirement documents; they define reward rubrics, personality baselines, and model behavior guidelines.
  • Frontier AI research labs treat model development like Snape's potions class: teams add specific ingredients, but outputs remain unpredictable until tested.
  • Product managers must translate messy qualitative user feedback directly into training data specs and evaluation datasets for research scientists.
  • In large organizations building multiple AI products, the PM role centers on driving consensus and turning experimental chaos into operational order.

Software Gets Cheap, Taste Gets Expensive

When code is cheap to generate, software floods the market. Writing code used to be the bottleneck that stopped bad ideas from reaching production. Now anyone can prompt an application into existence over a weekend. That shift creates a new problem: when building software is effortless, knowing what to build becomes the only thing that separates winners from noise.

Tulsee Doshi sees this dynamic play out daily as Head of Product for Google's frontier models. In her words: “When you can build anything and you can build anything so easily, building something great actually is now a unique value add. And taste is now really important. Someone who can come in and actually really pull in what is the signal from the noise.” (play from 20:26)

For years, critics predicted AI would eliminate product management. If engineers can converse directly with coding assistants, why keep a translator in the room? The reality inside frontier labs is the opposite. When an organization can generate fifty features a day, deciding which forty-nine to throw away requires high-conviction judgment. Instead of scheduling sprints and grooming backlogs, PMs must now exercise product taste to separate real user intent from vanity prompts. Doshi points out that inside Google, where dozens of teams build on top of Gemini, the PM must “bring people together and drive consensus and turn chaos into order.” (play from 21:06)

Frontier Models Are Snape's Potions Class

Traditional software engineering is deterministic. If you write a database query, you know what rows return. Frontier AI models do not work that way. You train on billions of tokens, tune weights, adjust reward functions, and wait to see how the system reacts.

“The model isn't predictable,” (play from 0:00) Doshi explains. “It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising.” (play from 0:00)

Because models behave unpredictably, a PM cannot just hand off a static product requirements document (PRD) to an engineering team. That workflow is dead. Doshi's team at Google focuses directly on model behavior. They define what Gemini should sound like, how it handles sensitive questions, and where its personality lands on the spectrum between concise and conversational. As Doshi notes, “Your job is no longer just writing down a set of requirements. Your job is actually going one step further to have some heft to that, whatever format that heft takes.” (play from 50:48)

That heft means translating user feedback into the actual math that trains the system. When users complain that Gemini feels defensive or overly verbose, Doshi's PMs do not file a bug ticket. They define the reward rubrics. They build evaluation suites and curate training examples that guide reinforcement learning teams. The PM becomes the voice that defines what good looks like in behavioral terms.

What to Do With This

Stop writing feature PRDs for probabilistic features. Tomorrow morning, open your product's AI prompt templates or system instructions and write a 10-row evaluation rubric. For each row, specify a realistic user prompt, the failure mode your model currently exhibits, and the exact tone and reasoning structure of an ideal output. Hand those 10 concrete examples to your engineering team as the baseline test for your next deployment.