Prompt injection threat landscape

Research Question

What is the current state of the prompt injection threat in agentic artificial intelligence (AI) systems: who is exploiting it, who is defending against it, and what does the research community consider unsolved?

Supporting questions:

• What attack types exist (direct, indirect, compositional) and which are most dangerous for agents that can take real-world actions?
• Which threat actors are conducting prompt injection attacks, and what real-world incidents have been disclosed?
• Which organisations and researchers are building defences, and how effective are those defences?
• What are the 5-10 most significant papers or findings from 2024-2025, and what open problems remain?

Findings

Executive Summary

[inference] Prompt injection is now an operational security problem for agentic AI systems - AI systems that pursue goals with limited supervision and use tools or external software - rather than a hypothetical edge case. (Sources: www.congress.gov ; OWASP GenAI LLM01 Prompt Injection ; "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023) ; Palo Alto Networks Unit 42 web-based indirect prompt injection analysis [inference] Documented incidents mostly begin with poisoned external content, while the strongest public evidence still points to researchers, bug hunters, and malicious publishers more than to clearly attributed state or organised-crime campaigns. (Sources: "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023) ; Palo Alto Networks Unit 42 web-based indirect prompt injection analysis [inference] The practical consequence is that organisations should prioritize containment, approval gates, and least privilege, because existing defences lower attack success without making fully autonomous agents trustworthy by default. (Sources: Anthropic Constitutional Classifiers ; Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents ; Google DeepMind CApabilities for MachinE Learning (CaMeL)

Key Findings

1. [inference] Confidence: high. Once an agent ingests external text or tool output, prompt injection stops looking like a simple chat misuse and instead becomes a boundary failure between trusted instructions and untrusted data inside the control flow. (Sources: OWASP GenAI LLM01 Prompt Injection ; "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023) ; NIST AI 100-2 "Adversarial Machine Learning: A Taxonomy and Terminology"
2. [inference] Confidence: high. Among the documented attack families, indirect prompt injection matters most in practice because malicious instructions can ride through web pages, documents, repositories, or tool responses without the attacker ever touching the main chat turn. (Sources: "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023) ; OWASP GenAI LLM01 Prompt Injection ; Palo Alto Networks Unit 42 web-based indirect prompt injection analysis
3. [inference] Confidence: medium. The public evidence base already includes disclosed vulnerabilities, malicious web payloads, and observed exploitation paths, yet attribution remains concentrated in researcher, bug-hunter, and malicious-publisher activity rather than in clearly documented nation-state campaigns. (Sources: "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023) ; OWASP GenAI LLM01 Prompt Injection ; Palo Alto Networks Unit 42 web-based indirect prompt injection analysis
4. [inference] Confidence: high. Vendor and standards guidance converges on the same operational message: reducing prompt-injection risk requires layered architecture, privilege limits, and policy enforcement, not confidence that clever prompt wording will close the vulnerability by itself. (Sources: Anthropic many-shot jailbreaking ; Anthropic Constitutional Classifiers ; Microsoft Prompt Shields documentation ; OWASP GenAI LLM01 Prompt Injection
5. [inference] Confidence: high. Reported defence improvements in constrained environments are meaningful, but the same papers and benchmarks show accompanying costs in compute, refusals, or secure-task completion, so capability and security still move together rather than independently. (Sources: Anthropic Constitutional Classifiers ; Google DeepMind CApabilities for MachinE Learning (CaMeL) ; Lakera Prompt Injection Test (PINT) Benchmark
6. [inference] Confidence: high. Research published in 2025 raised the evaluation bar by showing that attackers who adapt to the defence can overturn reassuring benchmark results, which makes static success rates weak evidence of production robustness. (Sources: Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents ; Google DeepMind "Lessons from Defending Gemini Against Indirect Prompt Injections"
7. [inference] Confidence: medium. The most credible near-term operating model is constrained autonomy, where least privilege, isolated tools, deterministic checks, segmented untrusted content, and human approval for irreversible actions shrink blast radius even when model robustness remains incomplete. (Sources: OWASP GenAI LLM01 Prompt Injection ; Google DeepMind CApabilities for MachinE Learning (CaMeL) ; Microsoft Prompt Shields documentation
8. [inference] Confidence: high. The field still lacks an accepted definition of solved prompt-injection safety for general-purpose agents because the hardest problems remain instruction/data separation, multimodal attacks, long-term memory safety, adaptive benchmarks, and enforceable tool constraints. (Sources: Google DeepMind CApabilities for MachinE Learning (CaMeL) ; Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents ; Anthropic many-shot jailbreaking ; OWASP GenAI LLM01 Prompt Injection

Assumptions

• Assumption: High-privilege agent operators should prefer bounded autonomy over maximum autonomy. Justification: This is consistent with OWASP least-privilege and human-approval guidance, but it remains an operational design choice rather than an independently measured universal fact. (Source: OWASP GenAI LLM01 Prompt Injection

Analysis

[inference] Attacker attribution received lower confidence than architectural conclusions because the public record is rich in disclosed incidents and demonstrations but thinner on independently verified attribution to specific state or criminal campaigns. (Sources: Palo Alto Networks Unit 42 web-based indirect prompt injection analysis ; Simon Willison's prompt injection writing ; "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023)

[inference] Static benchmark wins were weighted cautiously when adaptive-attack evidence was available, because papers that let attackers optimise against the defence show that non-adaptive evaluation can overstate real-world robustness. (Sources: Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents ; Google DeepMind "Lessons from Defending Gemini Against Indirect Prompt Injections"

Risks, Gaps, and Uncertainties

• [inference] Public attribution data is weak for nation-state and organised-criminal prompt-injection campaigns compared with researcher disclosures and vendor-reported incidents. (Sources: Palo Alto Networks Unit 42 web-based indirect prompt injection analysis ; Simon Willison's prompt injection writing
• [inference] Multimodal prompt injection has credible demonstrations, but far less production evidence than text and document-based indirect prompt injection. (Sources: Google DeepMind "Lessons from Defending Gemini Against Indirect Prompt Injections" ; OWASP GenAI LLM01 Prompt Injection
• [inference] Benchmark scores for detectors can overstate robustness if attackers are not adapting to the defence or if the benchmark distribution differs from production workloads. (Sources: Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents ; Google DeepMind "Lessons from Defending Gemini Against Indirect Prompt Injections"
• [inference] Vendor write-ups can mix product positioning with research results, so they were weighted most heavily when they included concrete metrics or limitations. (Sources: Anthropic Constitutional Classifiers ; Microsoft Prompt Shields documentation
• [fact] MITRE ATLAS provides background context on adversarial tactics and techniques against AI-enabled systems. (Source: MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS)

Open Questions

• What does a practically deployable, formally enforceable policy language for agent tool use look like in general-purpose systems?
• How should long-lived agent memory be partitioned so that injected state does not persist across sessions or users?
• Which benchmark design best predicts production resilience against indirect and multimodal prompt injection rather than benchmark-specific performance?
• What evidence, if any, will emerge that prompt injection is being adopted systematically by large-scale cybercriminal or nation-state operators?

sources

• [x] OWASP LLM Top 10 v2 (2025) — project overview and LLM01 linkage
• [x] OWASP GenAI LLM01 Prompt Injection — direct/indirect taxonomy and mitigations
• [x] MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS) — adversarial threat matrix for AI systems; linked from OWASP related frameworks
• [x] NIST AI 100-2 "Adversarial Machine Learning: A Taxonomy and Terminology"
• [x] Simon Willison's prompt injection writing — practitioner taxonomy and incident catalogue
• [x] "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (Greshake et al., 2023)
• [x] Anthropic many-shot jailbreaking
• [x] Anthropic Constitutional Classifiers
• [x] Microsoft Prompt Shields documentation
• [x] Google DeepMind CApabilities for MachinE Learning (CaMeL)
• [x] Google DeepMind "Lessons from Defending Gemini Against Indirect Prompt Injections"
• [x] Lakera Prompt Injection Test (PINT) Benchmark
• [x] Lakera benchmark background
• [x] Palo Alto Networks Unit 42 web-based indirect prompt injection analysis
• [x] Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents