Key Takeaways

  • Julius AI founder Rahul Sonwalkar released Deep Finance to test whether artificial intelligence models can execute private equity and M&A deals.
  • The benchmark drops models into messy virtual data rooms containing unstructured emails, conflicting financial filings, and raw audits to generate DCF models, LBO models, and investment memos.
  • Coding-focused models consistently fail basic financial checks, showing that writing clean software does not guarantee balance-sheet accuracy.
  • To prevent testing artifacts, Julius ran evaluations directly inside native command-line tools, pairing OpenAI models with Codex and Anthropic models with Claude Code.
  • Thinking Machines' open-source Inkling 2 model beat Nvidia Nemotron, finishing as the top American open-source model and second globally behind Kimi.

Virtual Data Rooms Break Standard Coding Logic

Most artificial intelligence benchmarks feed a model a clean prompt and expect an isolated answer. Real Wall Street transactions look nothing like that. When an analyst works on an acquisition or a buyout, they enter a virtual data room filled with hundreds of messy files. They dig through internal emails, reconcile financial filings that contradict each other, and verify numbers against raw audit papers.

Sonwalkar built Deep Finance to recreate that exact grind. As Sonwalkar explained, “The goal is to see how good models do on real world big deals that you would see in investment banking and private equity.” (play from 1:12:15) The models must absorb the unstructured mess and build full discounted cash flow models, LBO models, and investment memos.

The test exposes an immediate split between software engineering ability and financial logic. Engineers often assume that a model that writes great Python or SQL can easily build a financial model. The benchmark results prove the opposite. Models tuned purely for software development consistently fail financial sanity checks. A script can run without syntax errors and still generate a buyout model with broken debt paydown schedules, circular reference errors, or mathematically absurd revenue projections. Code tests check for functional execution; financial deal models test whether an economic narrative matches strict accounting constraints.

Native Tools and the Inkling 2 Result

Benchmark design often corrupts benchmark results. When evaluation platforms wrap models in synthetic testing wrappers, they create artificial constraints that skew the outcome. Julius AI avoided this by running models inside their natural operating environments. “We use the native CLI,” (play from 1:15:38) Sonwalkar noted, "so for OpenAI models we use Codex, for Claude models we use Claude Code."

Evaluating models through their native tools revealed unexpected rankings among open weights. The standout performer was Inkling 2, an open-source model from Thinking Machines. In Deep Finance runs, Inkling 2 outperformed established frontier models and surpassed enterprise-backed alternatives.

“When we ran their model on our eval, we found that it's actually the best American open source model,” (play from 1:17:34) Sonwalkar said. “And second best open-source model overall. After Kimi.” (play from 1:17:41) He added that seeing Inkling 2 perform “even better than Nemotron from Nvidia” (play from 1:18:58) highlights how specialized research labs are challenging tech giants in domain-specific tasks. While Chinese open-source models like Kimi still lead the global open-source category in financial reasoning, Inkling 2 proved that American open weights can handle long-horizon logic across conflicting documents.

What to Do With This

Stop evaluating model capabilities using clean API calls and generic coding puzzles. If you are building workflows for complex business operations, construct an internal test set of ten raw, messy historical files from your company's own records, complete with discrepancies and bad formatting. Run the models through their native command-line tools to produce an end-to-end operational deliverable, and grade them strictly on balance-sheet integrity rather than whether the code ran without errors.