Key Takeaways
- Scott Kirby uses customer surveys exclusively to audit past execution, never to decide future product strategy.
- During the pandemic, airline industry surveys reported that travelers were terrified and planned to avoid flying indefinitely, leading competitors to mothball fleets while United prepared for demand to return.
- Industry forecasters stick inside two standard deviations of historical data, which Kirby argues destroys value because outsized returns only exist in the statistical tails.
- John Collison observes that Stripe regularly deprograms engineers and product managers who outsource independent product judgment to metric dashboards.
- Kirby's operational rule matches Apple: fielding a survey takes more calendar time than shipping the product to reality and seeing whether customers pay.
The Two Standard Deviations Trap
When COVID grounded global aviation in 2020, industry consultants and airline executives sent out millions of customer surveys. The results showed absolute panic. Travelers reported that they were terrified of crowded cabins, planned to cut corporate trips forever, and would avoid vacation travel for years.
Most airline leadership teams looked at those survey dashboards and mothballed their fleets. Scott Kirby did the opposite. He recognized that surveys capture transient emotion, not future human behavior. People in crisis extrapolate their current dread into an indefinite future.
“Surveys are the absolute worst way to figure out what's going to happen in the future,” Kirby said on Cheeky Pint. “Because the reason travel experts thought travel... They did surveys, and people were terrified.”
Kirby pointed out that forecasters fail whenever conditions break historical patterns. “Experts live in the world of two standard deviations. They force-fit everything into the things that they've already seen. When things happen outside two standard deviations, when the tails happen, which is where you create all the value.”
United bet against the survey results. They ordered hundreds of narrowbody and widebody jets while competitors canceled deliveries. When lockdowns ended, demand snapped back faster than the industry could supply seats. The survey data had been completely wrong about future behavior because it was accurate only about momentary fear.
Looking Backward Versus Building Forward
Kirby maintains a strict dividing line for user research at United:
If United wants to know whether the gate check-in app crashed or whether an in-flight meal tasted terrible, surveys provide clean operational signal. But if United needs to decide whether to install SpaceX Starlink Wi-Fi across their fleet, or in-house their software stack, asking customers what they think they will want is a waste of time. Customers have no context for products that do not exist yet.
John Collison sees the exact same dysfunction inside high-growth tech companies. “Definitely at Stripe, we find we need to deprogram people a bit because in tech, people love running things via metrics, and you can outsource your thinking to the dashboards and metrics.”
Product teams often hide behind survey results or metric dashboards because data feels safe. If an executive questions a feature, pointing to a survey slide deflects blame. But outsourcing judgment to user surveys slows execution down to a crawl.
“It takes more time to do a survey or a focus group than just do it,” Kirby said. “See if they like it. And if they don't, stop. And if they do, do more of it.”
What to Do With This
Audit your product roadmap tomorrow morning. If you have any roadmap item justified primarily by user survey responses or speculative focus groups, kill the research project and run a live pilot instead. Build a raw version of the feature in seven days, ship it to 5% of your active users, and track whether their actual usage matches their stated preferences.