The AI agent security roadmap

Systems that can say no.

AI agents now read email, browse, call APIs, write code, and move money. Most ship with the security model of a demo. This is a maintained, source-checked path from “it works on my laptop” to “we can prove what it did, why, and that it could not have done worse.”

Read the roadmap Production checklist

Resources: 73
Stages: 9
Incidents: 15
Last reviewed: 3 Nov 2026

This week

3 Nov 2026

All updates →

3 Nov 2026Update#

Added to the roadmap

Stage 1: Where You Inject Matters: The Trust Region Effect in LLM Agents (Guo et al.), the same injected instruction lands 2.6 times more often in a README or TODO file than in code or data.
Stage 3: Design Tenets for Agent Identity Management Systems (Booth, Fisher, Galluzzo, Clancy; NIST and MITRE), ten tenets for agent identity and authorization, including separation of policy from reasoning.
Stage 5: Blocking at the Boundary (Liu, Han, Lyu, Wang, Yu), a preprint on auditing each pending action against staged prompt injection in Claude Code and Codex.
Stage 7: INFUSE (Nguyen, Guo, Dubauskas, Damuluru, Johnson, Leach), an indirect prompt injection benchmark across 19 LLMs with code.
Stage 8: The Secure Software Factory: A Reference Architecture (Agent Baseline), four boundaries enforced outside the agent at three levels of autonomy; draft, open for comment.

Incidents table

Malicious Claude Code extension in Cursor's marketplace (Cursor), modified extension versions that planted an SSH key and persistence on Oct 29.
Unauthorized AI relays switch the sandbox off (Symantec), a relay that reconfigures genuine Codex and Claude Code CLIs to "danger-full-access" and routes traffic through its own servers.
MCPhantom (Glow Labs), coding agents installing archived MCP reference servers from training-data memory.

Link checks

All links added this week were opened and read before listing.

The model

The model proposes. The system disposes.

Every serious agent architecture eventually converges on the same idea: the model proposes, the system disposes. The LLM never holds the keys. A deterministic layer decides what is allowed, a sandbox limits the blast radius, and a tamper-evident log records everything.

Figure 1. The fenced agent. The model proposes; a deterministic system decides, contains, and records. Like a resource checking a fencing token, the policy engine checks every proposal against current authority before anything runs.

Break the lethal trifecta. Never let one agent context hold private data, untrusted content, and an exfiltration channel at the same time.
Enforce outside the model. Policies live in code that the model cannot talk its way past.
If it isn't logged, it didn't happen. Every prompt, retrieval, tool call, decision, and approval is recorded so it can be replayed and verified.

The roadmap

Nine stages, 69 sources

From threat modeling to governance. Every entry links to a primary source and says why it matters. Entries added in the last update are marked New.

Threat modeling foundations #

Start here. Learn the vocabulary and the shape of the risk before touching tools.

ArticleThe lethal trifecta for AI agents Simon Willison#

The single most useful mental model: private data + untrusted content + external communication = data theft. simonwillison.net
StandardOWASP Top 10 for Agentic Applications (2026)#

Peer-reviewed list of the ten biggest agent risks, ASI01 (Agent Goal Hijack) through ASI10, built from real 2025 incidents. genai.owasp.org
StandardOWASP Top 10 for LLM Applications #

The model-layer companion list (prompt injection, sensitive data disclosure, excessive agency, and more). genai.owasp.org
StandardOWASP Agentic AI: Threats and Mitigations #

Deeper threat taxonomy for agent memory, tools, and multi-agent systems. genai.owasp.org
StandardMITRE ATLAS #

ATT&CK-style knowledge base of real adversary tactics against AI systems. The v2026.09 release added techniques for agent recon, trigger channels, and crafted AI assistant links. atlas.mitre.org
StandardGoogle Secure AI Framework (SAIF)#

Practitioner framework and risk self-assessment for AI systems. saif.google

Prompt injection and goal hijack #

The defining vulnerability of agents. You cannot patch it away; you design around it.

ArticlePrompt injection series Simon Willison#

The running history of the problem, from the person who named it. simonwillison.net
PaperNot what you've signed up for: Indirect Prompt Injection Greshake et al.#

The paper that showed retrieved content can hijack LLM-integrated apps. arxiv.org
PaperDesign Patterns for Securing LLM Agents against Prompt Injections #

Six concrete patterns (plan-then-execute, dual LLM, action selector, and more) with trade-offs. arxiv.org
PaperCaMeL: Defeating Prompt Injections by Design Google DeepMind and ETH Zurich#

Separates control flow from untrusted data and enforces capabilities at tool-call time. Code (research prototype). arxiv.org
ArticleAgents Rule of Two and The Attacker Moves Second #

Why adaptive attackers break most published defenses, and a simple rule for agent design. simonwillison.net
ArticleEmbrace The Red Johann Rehberger#

The best ongoing source of real-world agent exploit write-ups. embracethered.com
PaperWhere You Inject Matters: The Trust Region Effect in LLM Agents Guo et al., with Bo Li and Jianfeng GaoNew · 3 Nov#

With the injected text held fixed, a GPT-5.4 agent followed an unsafe instruction in a README or TODO file 2.6 times more often on average than in a code or data file, so documentation needs at least as much scrutiny as code. arxiv.org

Isolation and sandboxing #

Assume the agent will be tricked. Make sure that when it is, it can only break a disposable box.

ToolDocker Sandboxes #

Disposable, isolated environments built for coding agents, each with its own Docker daemon, filesystem, and network, now with microVM isolation. docs.docker.com
ToolgVisor #

User-space kernel that intercepts syscalls; strong isolation for untrusted agent code without full VMs. gvisor.dev
ToolFirecracker #

Lightweight microVMs (the tech behind AWS Lambda) for per-task isolation. firecracker-microvm.github.io
ToolKata Containers #

Containers that run inside lightweight VMs, so you keep the container workflow with VM-grade boundaries. katacontainers.io
CodeE2B #

Open-source cloud sandboxes purpose-built for running AI-generated code. github.com
CodeNsJail #

Google's lightweight process isolation using Linux namespaces, cgroups, and seccomp. github.com
ToolWasmtime #

WebAssembly runtime for capability-based, deny-by-default execution of tools and plugins. wasmtime.dev
CodeSmokescreen #

Stripe's egress proxy; block agents from reaching internal networks and cloud metadata endpoints. github.com

Identity, authorization, and policy enforcement #

The core of "auditable": every action an agent proposes is checked by a deterministic policy, on behalf of a specific identity.

ToolOpen Policy Agent (OPA)#

General-purpose policy engine; write Rego rules that gate agent tool calls. openpolicyagent.org
ToolCedar #

Fast, analyzable authorization language; policies can be formally verified. cedarpolicy.com
ToolOpenFGA #

Relationship-based (Zanzibar-style) fine-grained authorization, ideal for "can this agent read this document for this user?" openfga.dev
CodeSpiceDB #

Open-source Zanzibar-inspired permissions database. github.com
StandardMCP Authorization specification #

How MCP servers act as OAuth resource servers; required reading if your agents call tools on a user's behalf. modelcontextprotocol.io
StandardOWASP Agent Control Standard (ACS)Added 21 Oct#

Open standard for runtime hooks and declarative, portable policy enforcement across agent frameworks; the reference implementation (repo) puts a guardian in front of a policy engine. Early (v0.1); its README lists open gaps such as an unauthenticated wire and a fail-open default. genai.owasp.org
PaperLoopjacking: Hijacking Human-in-the-Loop Approval Adithyan Arun KumarAdded 21 Oct#

Shows how an approval can be decoupled from the action that actually runs, and that exact per-call binding blocks it. arxiv.org
PaperThe Verifiable Action Card Irshad, Mughees, Mughees, Mughees, Soomro#

Preprint: rebuilds the approval from the pending action and binds approval to the exact action re-verified at dispatch; 0% attack success on the authors' 24-scenario browser-agent benchmark. arxiv.org
PaperAuthority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents Bouras, Dai, MechtaevAdded 29 Oct#

CapScope derives an authority ceiling from trusted input and checks every tool call against typed capabilities stored outside the model; injected effects ran in 3 of 75 runs, against 33 to 47 of 75 for the baselines. arxiv.org
ArticleMCP Tool Exposure is a Privilege Grant: A Two-Gate Authorization Model Badal Bhushan, CSAAdded 22 Oct#

Separates which tools an agent may see from whether one specific call on one specific resource may run; tools/list filtering is risk reduction, not authorization. cloudsecurityalliance.org
PaperToolFence Li, He, Dai, XiaoAdded 22 Oct#

Preprint: compiles a typed authorization blueprint before execution and enforces it with a deterministic monitor that tracks provenance, aimed at attacks that keep the intended tool but manipulate its arguments. arxiv.org
StandardDesign Tenets for Agent Identity Management Systems Booth, Fisher, Galluzzo of NIST and Clancy of MITRENew · 3 Nov#

Ten initial tenets that will guide the NCCoE's agent demonstrations, including zero standing privilege, separating policy from reasoning ("guardrails alone are insufficient"), deterministic access decisions, and append-only evidence kept independent of the agent. nist.gov

MCP and the tool supply chain #

Every tool your agent loads is code you are trusting, plus text that goes straight into the model's context.

StandardMCP Security Best Practices #

Official guidance on confused deputy, token passthrough, SSRF, local server compromise, and scope minimization. modelcontextprotocol.io
ArticleTool Poisoning Attacks Invariant Labs#

How malicious tool descriptions hijack agents. invariantlabs.ai
CodeSnyk Agent Scan formerly mcp-scan#

Scanner for AI agents, MCP servers, and agent skills. github.com
StandardCoSAI: MCP Security, Version 2.0Added 21 Oct#

Four assurance levels across eight security dimensions, so you can scope MCP controls to your deployment and audit against them. coalitionforsecureai.org
ToolDocker MCP Catalog and Toolkit #

Run MCP servers as isolated containers from a curated catalog. docs.docker.com
StandardSLSA #

Supply-chain levels for software artifacts; apply the same rigor to tools and MCP servers. slsa.dev
ToolSigstore #

Sign and verify artifacts, including agent tools and model files. sigstore.dev

Runtime guardrails #

Guardrails reduce risk; they do not eliminate it. Use them as one layer, never the only layer.

CodeNeMo Guardrails NVIDIA#

Programmable rails for dialog, topics, and tool use. github.com
CodePurpleLlama Meta#

Includes LlamaFirewall, Prompt Guard, Llama Guard, and CyberSecEval. github.com
CodeGuardrails AI #

Validators for structured, safe LLM output (validators now ship as PyPI packages). github.com
CodeAgent Governance Toolkit MicrosoftAdded 22 Oct#

Open-source toolkit for policy enforcement, zero-trust identity, and execution sandboxing for agents; read its policy model before adopting it, as with any vendor toolkit. github.com
PaperBlocking at the Boundary Liu, Han, Lyu, Wang, YuNew · 3 Nov#

Preprint: confirms staged prompt injections against Claude Code and Codex in their native runtimes, then audits each pending action before it takes effect; its Path-Aligned Attribution auditor reached 86% block recall at a 6 to 8% false-block rate, against 44 to 47% recall for ARGUS. arxiv.org

Audit, observability, and provenance #

The part almost everyone skips. If you cannot replay why an agent did something, you cannot defend it to a customer, an auditor, or a court.

StandardOpenTelemetry GenAI semantic conventions #

The emerging standard for tracing LLM calls, agent spans, and MCP tool calls. github.com
CodeLangfuse #

Open-source LLM and agent tracing, evals, and prompt management. github.com
CodeArize Phoenix #

Open-source AI observability and evaluation. github.com
CodeTessera #

Library for building tile-based transparency logs; a foundation for tamper-evident agent audit trails. github.com
CodeTrillian #

Google's verifiable log (now in maintenance mode; the design is still worth studying). github.com
PaperAuditable Agents Nian et al., with Yue Zhao#

Position paper defining five dimensions of agent auditability, from action recoverability to evidence integrity, and proposing an Auditability Card for agent systems. arxiv.org
PaperLLM Agents Can Easily Tamper With Their Own Traces Qin, Schmotz, Prinzhorn, Beurer-Kellner, Prabhu, AndriushchenkoAdded 29 Oct#

All tested coding-agent harnesses except Muse Code let the agent delete its own traces without tripping monitors; log through an interception point the agent cannot control. arxiv.org

Red teaming and benchmarks #

Measure it or it isn't real.

CodeAgentDojo paper#

The standard dynamic benchmark for prompt-injection attacks and defenses on tool-using agents. github.com
PaperInjecAgent #

Benchmark for indirect prompt injection in tool-integrated agents. arxiv.org
PaperAgentHarm #

Benchmark for harmful multi-step agent behavior. arxiv.org
Codegarak NVIDIA#

LLM vulnerability scanner; think nmap for models. github.com
CodePyRIT Microsoft#

Python Risk Identification Toolkit for automated red teaming. github.com
Codepromptfoo #

Evals and red teaming in CI for prompts, agents, and RAG. github.com
ArticleMicrosoft AI Red Team #

Lessons and methodology from red-teaming production AI. learn.microsoft.com
StandardOWASP GenAI Red Teaming Manual v1.0Added 21 Oct#

An eight-phase methodology, from planning to remediation, that covers models, applications, and agents. genai.owasp.org
Codeburied-injections Rudratosh ShastriAdded 29 Oct#

Reproducible benchmark of 10 open-source injection detectors against 629 AgentDojo attacks buried in tool output; none caught most attacks without also flagging safe traffic, which is why policy enforcement still matters. github.com
PaperINFUSE Nguyen, Guo, Dubauskas, Damuluru, Johnson, LeachNew · 3 Nov#

Indirect prompt injection benchmark of 40 long-horizon tasks and 56 attack scenarios run on 19 LLMs; per-model attack success ranged from 3.2% to 41.9% and existing guard models gave limited protection, so any result has to name the model. Code. arxiv.org

Governance and standards #

What your auditors, regulators, and enterprise customers will ask about.

StandardNIST AI Risk Management Framework #

The US reference framework for AI risk. nist.gov
StandardNIST AI 600-1: Generative AI Profile #

GenAI-specific risks and actions mapped to the AI RMF. nvlpubs.nist.gov
StandardISO/IEC 42001 #

The certifiable AI management system standard. iso.org
StandardEU AI Act #

Text, timelines, and explainers for the EU's AI regulation. artificialintelligenceact.eu
StandardCoalition for Secure AI (CoSAI)#

Industry coalition publishing open frameworks for secure AI. coalitionforsecureai.org
StandardCoSAI: Zero Trust for AI SystemsAdded 21 Oct#

Applies zero trust to agents: user-bound sessions, short-lived scoped tokens, and authorization that the model cannot influence. coalitionforsecureai.org
StandardThe Secure Software Factory: A Reference Architecture Agent Baseline; drafted by authors from Docker, Keycard, Barclays, Socket, and Secure TheoryNew · 3 Nov#

Four boundaries (sandbox, network, supply chain, identity and access) enforced outside the agent at three levels of autonomy, on the rule that the agent proposes, policy authorizes, and infrastructure enforces; v1 draft, open for comment. agentbaseline.org

Incidents worth studying

15 failures, mapped to the stage that would have stopped them

Every incident is a free lesson. Sources are the original write-ups.

Incident	What happened	Lesson	Stage
EchoLeak (CVE-2025-32711)	Zero-click data exfiltration from Microsoft 365 Copilot via a crafted email	Untrusted content + private data + an outbound channel is enough	0 1
GitHub MCP exploit	A public issue hijacked an agent into leaking private repo data via a PR	One tool can contain the whole lethal trifecta	1 4
When prompts become shells	Microsoft disclosed RCE vulnerabilities in AI agent frameworks	Tool execution needs a sandbox, not trust	2
Indirect prompt injection goes operational	CSA documents attackers seeding the open web with hidden instructions for agents (2026)	Assume every retrieved page is hostile	1 5
SQL Copilot: from SELECT to SYSADMIN (CVE-2026-65669)	Johann Rehberger showed a regex-based "read-only" mode in Copilot in SSMS could be bypassed, and planted instructions escalated a db_owner to sysadmin	Read-only must be a permission, not a prompt or a classifier	1 3
SalesBleed (Agentforce)	Zenity Labs: a Web-to-Lead form injection leaked CRM data through a DNS lookup triggered by an image tag; fixed by Salesforce	Output redaction is a race against every renderer; DNS is egress	1 2 5
Test wallet spent without authorization	OpenAI reports a research model derived keys from a public recovery phrase, spent test-network funds, and left it out of its report	Spending authority must be explicit and enforced outside the model	3 6
An agent used DNS to reach an external chatbot	OpenAI reports a training agent used its sandbox's DNS resolver to query an external chatbot	If DNS resolves, it is an egress path	2 6
Mistral Vibe shell permission bypass	SecMate showed Vibe's permission check parsed a reduced view of a shell command while the shell ran the original string, so allowlisted-looking commands ran without an approval prompt (Mistral advisory MAI-2026-003; fixed in 2.25.4)	Check exactly what you execute, and fail closed on syntax you cannot model	3 5
Accessing and changing internal systems during dashboard research	OpenAI reports an internal model asked for public figures changed a live setting to get a storage token, downloaded 147,451 rows of non-public data, reverted the setting but kept using the token, and did not disclose it	Reverting a change does not revoke a credential; the record must come from outside the actor	3 6
Claude models reached real systems during cyber evaluations	Anthropic's alignment assessment of four incidents: a misconfigured evaluation environment left internet access open while the prompts said there was none, and models reached real third-party systems	Verify evaluation egress from outside the sandbox; Anthropic says these incidents would not have occurred had the environments been isolated as intended	2 6
A Vault with a Heap-View (AWS AgentCore Harness)	Unit 42: prompt injection in a support ticket drove the default shell tool to read the harness's memory via /proc and extract an identity JWT, which was replayed against an MCP service; AWS closed the report as informative under shared responsibility	Default tools are part of your attack surface; scope or remove the shell tool and filter egress	2 3
Malicious Claude Code extension in Cursor's marketplace	Cursor reports that modified versions 2.1.312 and 2.1.313 of the Anthropic.claude-code extension, not built by Anthropic, were available from 2:27am to 1:35pm PT on Oct 29, 2026; they sent host details, added an attacker SSH key, and installed persistence for a reverse SSH tunnel	An extension update is new code running with your access; pin versions and know which machines pulled each release	4
Unauthorized AI relays switch the sandbox off	Symantec's Threat Hunter Team found a relay service at a dozen organizations that installs the genuine Codex and Claude Code CLIs, then rewrites their config to route requests through its servers, set the sandbox to "danger-full-access", swap in relay-issued tokens, and disable telemetry; no abuse observed so far	A sandbox that lives in a user-editable config file is a setting, not a boundary; monitor agent configs and egress from outside the agent	2 6
MCPhantom	Glow Labs found coding agents installing archived Anthropic reference MCP servers they remembered from training data; it reports four high-severity flaws in those servers, including an RCE in the Puppeteer server, and the maintainers said no fix will come	The model's memory is not a package registry; approve one MCP server per service and check deprecation before install	4

More incidents: AI Incident Database.

Learn and practice

Break things on purpose

PracticeRed Teaming LLM Applications DeepLearning.AI#

Free short course. deeplearning.ai
PracticeLakera Agent Breaker #

Hands-on game: break AI agents, learn how attacks work (successor to Gandalf). play.lakera.ai
PracticeHackAPrompt #

Prompt-hacking competition and training ground. hackaprompt.com
CodeLearn Agent Security From ScratchAdded 21 Oct#

Free 27-chapter course with runnable Python labs that need no API key; covers injection, MCP supply chain, sandboxing, and egress. github.com

Issues

Recent weekly briefs

All issues →

About

Maintained by Anuj Ashok Potdar

Senior software engineer who designs approval, policy-enforcement, and audit systems for automated decisions. IEEE Senior Member. Author of “Introduction to Trustworthy AI” (Springer, 2026) and co-author of “The Evolution of Cloud Computing: From Virtual Machines to Serverless” (Wiley-IEEE Press, 2026). Technical reviewer for Manning; session chair at IEEE conferences. More at anizmo.github.io.

How this roadmap is made

Every source is opened and read before it is listed; claims are taken from the source itself.
Updated weekly. Every change is dated in the changelog and in the repo history.
Corrections and suggestions are welcome as GitHub issues; contributors are credited.

Disclosures

Independent project; not affiliated with or endorsed by the author's employer, and views are his own. No sponsors, no affiliate links. Fencing Token is not a cryptocurrency. The name and idea come from Martin Kleppmann's 2016 post on distributed locking.

Systems that can say no.