Key Takeaways

  • The Handshake AI team built ATLAS Finance to test frontier AI models against realistic 15-to-30-hour investment banking assignments spanning Outlook, shared drives, and complex financial workbooks.
  • In one benchmark test, an AI agent built a leveraged buyout model with over 2,600 live formulas but botched a 338(h)(10) tax election by improperly subtracting existing goodwill from the asset basis.
  • Fewer than 50 bad cells out of 2,600 rendered the entire valuation workbook unusable, because formula errors compound across connected accounting statements.
  • Sahil Bhaiwala revealed that when Handshake surveyed hundreds of Wall Street bankers, the vast majority said they would rather build a model from scratch than audit an AI output that is 95% accurate.

The Compounding Trap of 2,600 Formulas

AI labs love to celebrate 95% benchmark accuracy. In consumer chat, a 5% error rate is charming. In Wall Street financial engineering, it is catastrophic.

Collin Schweiker demonstrated this failure mode through ATLAS Finance, an evaluation environment designed to replicate junior banker workloads. The task was straightforward for a top private equity associate: build an integrated leveraged buyout (LBO) model from corporate documents, emails, and financial history. The agent actually did impressive heavy lifting. It wrote clean Python code to construct a workbook containing over 2,600 live formulas.

Then it hit the tax code.

When handling a Section 338(h)(10) tax election, the agent had to calculate the new asset basis. It tripped over corporate accounting rules. “The big issue is that when it was building its model in Python, it didn't realize that it was making a decision on the existing goodwill,” (play from 1:25:56) Schweiker explained. The agent improperly subtracted existing goodwill from the asset basis.

Out of 2,600 calculations, fewer than 50 cells contained logic errors. That sounds like an A grade on any standard software benchmark. In an integrated model, however, tax basis dictates depreciation, depreciation changes operating earnings, operating earnings alter debt paydown schedules, and debt balances determine interest expense. The balance sheet fell out of balance. Internal rates of return drifted off course.

“If it's 95% right, it's 100% wrong,” (play from 1:15:17) Schweiker said. “It takes just as much time, if not more time, to fix a flawed output as it is to just build from scratch or use a template.” (play from 1:15:20)

Why Bankers Refuse to Audit Agent Work

Founders often assume that giving an analyst a half-finished draft saves 50% of their time. Handshake tested that assumption against reality.

“For this benchmark for example we interviewed some hundreds of bankers and one question we asked I thought was really interesting is would you rather start from this output or not,” (play from 1:39:04) Sahil Bhaiwala noted. “The vast majority said that they would have rather started from scratch.” (play from 1:39:14)

Auditing someone else's spreadsheet formulas is grueling work. When an associate builds a model, they know the lineage of every link. When an autonomous model hands them 2,600 formulas, they have to trace every single calculation cell by cell to ensure a silent hallucination did not corrupt the debt schedule. Checking 2,600 formulas takes longer than building the sheet from an established firm template.

Schweiker summarized the dilemma bluntly: “5% off the true number doesn't sound like a lot but it compounds over time and particularly in finance where everything is data driven and numberbacked becomes a huge issue. So again, it's functionally unusable.” (play from 1:30:19)

If an autonomous agent cannot verify its own logical chain with zero accounting errors, human operators will discard the entire deliverable. For enterprise workflows with dense interdependencies, near-miss intelligence provides negative commercial value.

What to Do With This

Open your product roadmap this week and find any feature where your agent produces interdependent calculations or multi-step logic. If you currently display an end-to-end draft for human review, kill that flow; replace it with modular sub-step generation where the user verifies inputs before downstream formulas run.