Key Takeaways
- Factual hallucinations in finance are fading rapidly; the real barrier to enterprise adoption is human aesthetic judgment and subjective taste.
- OpenAI lead Joseph Kim finds that humans dramatically outperform LLM judges at evaluating visual and structural artifacts like presentation slides.
- Training subjective quality relies on pairwise human comparisons feeding a reward model rather than programmatic verification rules.
- In finance, model validity is often judged by feel, where analysts perturb inputs to test whether outputs react with intuitive proportions.
Hallucinations Are Fading, Taste Is Not
Most founders building vertical AI tools spend their cycles guarding against factual errors. They assume enterprise finance teams reject AI because they fear bad numbers. Joseph Kim, who leads ChatGPT for Financial Services at OpenAI, argues that this diagnosis is already stale. Factual errors are largely dropping off. The real hurdle is that models lack human aesthetic discretion.
When a senior associate reviews an analyst's deck, they do not just check whether the revenue numbers match the 10-K filing. They look at layout balance, information hierarchy, and whether an executive can digest the takeaway in three seconds. Software engineering faced this years ago with code reviews: code can compile, pass all unit tests, and still get rejected because it is unmaintainable, ugly, or poorly structured. Financial workflows carry identical unwritten standards. Until models master those standards, human analysts will keep rewriting the output from scratch.
The Feel of a Financial Model
Verifiable checks work when an objective truth exists. You can write an automated test to confirm whether an equation sums a balance sheet correctly. You cannot easily write a deterministic check for how a financial model behaves under stress.
Kim observed this dynamic directly when studying how institutional analysts assess models: “Something I learned about as I started working in finance is there's this notion of like the feel of the model, which is, you know, you kind of perturb the inputs and you look at the outputs.”
Experienced investors test models by pushing extreme numbers into the assumptions tab. They look at how margins flex, how working capital scales, and whether the curves feel realistic. This is an intuitive calibration built over thousands of hours of market exposure. Models trained solely to output factually correct calculations fail this qualitative test because they do not understand the implicit bounds of financial sanity.
Why Automated Evaluators Fail at Design
When teams try to train models on aesthetic tasks like slide generation, their first instinct is often to use another LLM as an automated judge. Kim found that this approach falls apart when evaluating subjective outputs.
Because language models struggle to grade design coherence, OpenAI relies on standard preference ranking. “The practical answer is you just train a reward model. You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better.”
If you want an AI system that produces presentation materials or models that finance executives actually accept, you cannot skip manual human evaluation. Automated synthetic benchmarks will tell you if the math works, but only human pairwise ranking will tell you if the deck looks ready for the boardroom.
What to Do With This
Take twenty outputs from your vertical AI product, whether they are slides, memos, or financial tables, and generate two competing layout variations for each. Put both versions side by side in a basic internal tool, hand the tool to your domain experts, and force them to pick the better one in under five seconds without explaining why. Feed those binary preference labels straight into your reward dataset to train taste before writing another prompt tweak.