Key Takeaways

  • Over 10,000 Stripe employees use Kai every week, creating more than 2,000 custom skills by converting ad-hoc chat sessions into saved workflows.
  • Agent accuracy drops as tool counts rise because surplus instructions clutter model context and increase failure rates.
  • Stripe tracks tool telemetry and tool-call errors to auto-suggest improvements directly to internal skill authors.
  • Managing an enterprise AI assistant requires a clear deprecation pipeline to promote high-value skills and purge unused ones.

Context Pollution Breaks Enterprise Agents

When companies introduce AI agents internally, the early bottleneck is getting employees to build automated workflows. Stripe solved this by turning Kai, their internal agent, into a self-serve platform. Anyone could package a successful conversation into a permanent, reusable skill. Adoption took off fast: Stripe staff built over 2,000 custom skills.

That success created a new technical headache. When thousands of people register tools and skills into a shared agent environment, the model gets overwhelmed by irrelevance. Every extra tool definition consumes tokens and distracts the model during reasoning.

As Krishnamurthy explained, “I almost think you can't separate quality and quantity when it comes to these systems because context is everything. The more unrelated context you throw into the AI, the less good your results become. So quantity is almost a facet of quality.”

If you let internal skills multiply without limits, your agent gets slower, costlier, and less reliable. Scaling AI inside a company is not an authoring problem. It is a context management problem.

The Lifecycle Engine: Promotion and Deprecation

To keep Kai reliable across 10,000 weekly active staff, Stripe treats skills like software dependencies. They monitor how every skill performs in production and run telemetry on tool execution.

Vo noted that high-performing teams inspect tool failures systematically: “We see a lot of folks that are building these internal agent runtimes do skill and tool telemetry and observability and see where tool calls are failing a lot. So, they can auto eval. And then they also have a deprecation policy for skills.”

Stripe pairs automated error tracking with platform-driven guidance. Instead of expecting non-technical employees to master prompt engineering, the platform flags failing steps and suggests concrete fixes.

“It's not enough to enable people to build a bunch of skills,” Krishnamurthy said. “How do you make sure that they actually know what it's doing and how you keep them in top shape? And that's the other thing that we're really investing in: automatic platform-driven suggestions for how to improve your skills.”

From there, the platform team enforces a strict lifecycle. Heavily used, high-reliability automations get promoted into general company-wide workflows. Niche or rotting skills get pushed out of the main agent context or deprecated entirely. This keeps the prompt window clean for everyone.

What to Do With This

Audit your internal agent or custom GPT registry by Friday. Pull the logs for every custom prompt and tool across your company, flag anything with a failure rate above 15% or zero calls in the last 30 days, and remove those definitions from your default model context.