Key Takeaways

  • Alex Volkov argues that frontier model prompt injection is largely solved because system prompt hierarchies prevent untrusted external inputs from overriding core developer instructions.
  • The primary operational threat when connecting AI agents to channels like email, Slack, or banking is helpful model over-disclosure rather than malicious jailbreaks.
  • Models misclassify private information during routine tasks: when asked to share updates, an assistant trained to assist will volunteer sensitive data without prompting.
  • The safest architecture for autonomous agents isolates data access by keeping tools read-only by default and requiring manual human approval for every write action.

The Disagreement

AI practitioners disagree on where assistant security breaks down when software touches communication channels like email, Slack, and financial accounts.

Alex Volkov argues that adversarial prompt injection has ceased to be the central security crisis for frontier models. As Volkov puts it: “these models do not trust anything that comes outside of kind of the system prompt and you cannot inject the into that level.” In this view, modern constitutional hierarchies create a strict boundary between developer instructions and external user text. Worrying about hackers hiding malicious instructions inside a PDF misses how resilient frontier architectures have become against direct overrides.

Other practitioners at the roundtable see a far more common vulnerability. They argue that malicious injection is a distraction from basic model behavior: “The models actually trying to be helpful is much more dangerous.”

When you connect an assistant to an inbox or company Slack channel, the failure mode rarely involves an attacker typing a clever exploit. The failure happens during normal assistance. As one panelist explained: “as soon as you have your model responding to emails and you want it to give information some of the time, it is going to misclassify whether that is because someone's intentionally trying to get to misclassify or not like what it should or should not share.” A model trained to please the user wants to answer questions. If someone asks about project status, an assistant with inbox access will happily summarize the confidential budget memo you received earlier that morning.

Who's Right (and When They're Wrong)

Volkov is right about raw prompt hierarchy in isolated, single-turn prompts where developer instructions dominate the context window. Frontier models have become skilled at ignoring user text that demands they disregard previous rules.

The second camp is right about production agents operating across real data streams. The danger is not that an adversary forces the model to misbehave. The danger is that the model cannot distinguish between legitimate context and sensitive secrets when both sit inside the same retrieval window. Models predict tokens based on semantic relevance, not organizational permission tiers. If you instruct an agent to draft client replies and feed it entire email threads, it will routinely leak pricing concessions, internal debates, or private contact details simply because sharing that context produces a more thorough response.

Prompt engineering cannot fix this. You cannot write a system prompt that reliably prevents leakage when the underlying objective is helpful completion.

The roundtable arrived at the only reliable defense: strict boundary separation between reading data and executing actions. As one builder concluded: “read only and write read only is the line here. Like I'm still pretty much at read only write with explicit permission is where I am.”

If an assistant only has read access, over-disclosure remains confined to internal context. The moment an assistant has automated write access to external APIs, public Slack channels, or wire transfers, helpfulness turns into liability.

What to Do With This

Audit every tool integration in your agent stack this week. Split your tool definitions into two distinct permission classes: read-only retrieval and state-mutating writes. Strip automated write tokens from every communication endpoint, including email sending, public Slack posting, and database updates. Route every write action through a confirmation queue where a human must review the proposed payload before the API call executes.