Key Takeaways

  • Arca operates as a full-service wealth management firm instead of a software vendor, giving leadership direct authority to mandate staff feedback on every agent output.
  • When an advisor overrides, alters, or cancels an agent action, Arca's product interface blocks them until they submit an explicit reason for the change.
  • Automated meta-agents audit execution traces every night, but human engineers review and commit all prompt edits by hand.
  • Letting models rewrite their own system prompts leads to overfitted, brittle logic where individual fixes hijack broader task instructions.
  • This continuous improvement cycle is structured through the Arca Closed-Loop Eval and Trace Improvement System.

The Arca Closed-Loop Eval and Trace Improvement System

Step 1: Mandatory Staff Feedback Capture

Require operational staff to submit a mandatory 'why' explanation whenever an agent recommendation or action is edited or canceled in the UI.

Step 2: Trace and Feedback Coupling

Pipe advisor feedback directly into a central datastore, tightly coupling the critique with the full execution trace of the agent that generated the output.

Step 3: Nightly Meta-Agent Trace Audit

Run an automated meta-agent overnight across production traces to identify error patterns, misleading tools, and missing knowledge in the operational knowledge base.

Step 4: Human-in-the-Loop Prompt Engineering

Require a human engineer to inspect meta-agent improvement proposals and synthesize prompt adjustments to avoid myopic, overfitted agent-written prompts.

Step 5: Direct Knowledge Base Curation

Provide domain experts a first-class product interface to curate, cite, and edit operational context files referenced by active agents.

When This Works (and When It Doesn't)

This system works for vertically integrated firms where AI assists internal knowledge workers on high-stakes tasks. In sectors like wealth management, insurance processing, and legal drafting, system operators sit inside your internal tool all day. That employment relationship gives you the operational authority to block their workflow until they document why an agent failed. As Singh put it: “That is another one of the advantages of us being the business and not a SaaS company is that we have the ability to have to require them to give us feedback.” He added: “All of that feedback gets piped into a data store that couples that feedback with the trace of the agent that produced it.”

This approach breaks down when selling standard enterprise software to third-party clients. External customers ignore optional text boxes, pick arbitrary dropdown answers, or churn if you block their progress with mandatory forms. The system also fails if you remove human oversight from prompt adjustments. Singh observed this failure mode during development: “Today we still have a human engineer in the loop before we actually ship a prompt change. I've noticed that agents will make really nasty prompt changes if you leave them do it. Every little fix becomes the whole prompt.”

What to Do With This

If you run an internal operations team using agents for document review or account setup, turn your interface into an eval engine this week.

First, update your operator interface on one high-volume task. When an operator rejects or modifies an agent-generated draft, replace the basic cancel button with a modal requiring a failure category and a one-sentence critique. Second, pipe that feedback straight to your database with the run trace ID attached. Third, add a dedicated context panel where operators can review and edit reference materials. As Singh noted: “On the left-hand side there's a tab called knowledge, which has the full operational context that our agents are using. And as they work on things, they're able to cite knowledge. And so, it creates this loop of like, oh, we notice that the knowledge is incorrect.” Have an engineer audit the coupled traces every Friday to update instructions by hand.