Key Takeaways

  • Joseph Kim, lead of ChatGPT for Financial Services at OpenAI, builds agent safety around existing corporate review hierarchies instead of novel alignment mechanisms.
  • In investment banking, output errors end careers, so OpenAI designs early tools around an explicit assumption that a human analyst reviews every output before it moves forward.
  • Future autonomous agent permissions borrow from Google's internal readability system, where engineers cannot push code without formal style and safety sign-off from accredited peers.
  • AI safety in regulated enterprise environments succeeds by mirroring the analyst-to-associate ladder: models hold read access and draft privileges while senior staff retain execution authority.

Why Wall Street Fears Model Autonomy

Finance has zero tolerance for hallucination. A bad cell in a DCF model or an erroneous debt covenant can cost millions of dollars and end someone's career. When enterprise software vendors sell AI to banks, they usually pitch automated efficiency. What bank risk committees hear is liability.

OpenAI's early approach to financial tooling sidestepped this obstacle by avoiding claims of autonomous perfection. The product design intentionally makes the user the final auditor. Kim explains the baseline setup: “The idea behind it is there's a a deep assumption that there's always a human in the loop, right?” That choice removed the regulatory burden from the foundation model. Kim notes: “And that made it very easy for us to to build a product where we could kind of pass off the responsibility of auditing review uh into the people's hands and into people's responsibilities.”

Google Readability Applied to Banking Chains

Moving beyond basic chatbots toward autonomous agents creates a permission problem. Giving an agent write access to ledgers, execution desks, or client portals terrifies risk officers. Kim solves this by copying standard corporate bureaucracy.

“There's this notion of read-write, even for people,” Kim notes. “Like, you're a new software engineer at a company, you oftentimes, you know, I was at Google, you need you need something called readability.” At Google, an engineer may understand algorithms, but they cannot commit code to production alone until a peer certified in that language's readability signs off on their style, tests, and formatting.

Investment banks rely on the same defense structure. “At least in the context of banking, which is where we kind of started, there's already this system in place for junior analysts doing a lot of the groundwork,” Kim says. “Their work gets passed off to associates. Associates are reviewing the work.” The model deck then moves to vice presidents, directors, and managing directors.

Instead of asking enterprise compliance teams to trust model evaluations or latent space filters, software makers should map software actions directly onto this org chart. “And some I would say something that people understand very intuitively is how to work with other people,” Kim says. “Our thesis is if we make the product very similar to how you work with your your co-workers, your interns, people around you, that is something that people can pick up and learn very quickly, very easily.” The agent acts like an entry-level analyst. It reads internal data, creates structured drafts, and remains technically incapable of committing an action without the next person in line.

What to Do With This

Audit your product's automated actions and restrict every agent API key to read and staging states. If your customer's company requires a vice president to sign off on an outgoing report or wire transfer, mirror that exact checkpoint in your software so the agent submits work to an associate review queue rather than executing on its own.