Adversarial Input Propagation Through Multi-Step Tool-Using LLM Systems

Adversarial Input Propagation Through Multi-Step Tool-Using LLM Systems: Error Amplification Across Verification and Strategy-Selection Phases

2026-05-18 · agentic-ai benchmarks-eval · medium · source → · wiki →

key claims

Indirect prompt injection succeeds because tool-using Large Language Model applications routinely concatenate retrieved content with user instructions, leaving models unable to reliably separate informational context from actionable commandsGreshake et al. (2023)Yi et al. (2025)Owasp (n.d.)
Once the loop's perception stage encodes a corrupted task state, strategy selection can remain internally coherent while optimizing for the wrong objective, so later tool calls inherit rather than repair the original semantic errorAnthropic (2026)Mitchell (2026)Greshake et al. (2023)
Tool use and persistent memory can extend adversarial corruption beyond one response by letting a locally plausible mistake change external state and, in documented cases, influence later sessionsMicrosoft (2025)Paloaltonetworks (n.d.)Greshake et al. (2023)
Generic self-verification by the same model class is not an independent safety layer, because intrinsic self-correction remains weak without external feedback and usually inspects the same contaminated context that produced the original mistakeHuang et al. (2024)Yi et al. (2025)Mitchell (2026)
Structured verification can recover some reasoning errors only when it introduces extra constraints such as explicit condition isolation, which shows that correction comes from added independence rather than from reflection aloneWu et al. (2024)Huang et al. (2024)
Unexpected environmental shifts and adversarial prompting are propagation-equivalent at the loop level when they both create observations that look familiar to the model while no longer preserving the environment's real causal structureAmodei et al. (2016)Ziegler et al. (2022)Mitchell (2026)
A useful formal approximation is that expected semantic error grows across repeated steps whenever path-dependent amplification and persistence exceed the error removed by independent correction signalsHuang et al. (2024)Greshake et al. (2023)Paloaltonetworks (n.d.)

Research Question

How do adversarial inputs or unexpected environmental shifts propagate error through a multi-step tool-using Large Language Model (LLM) system's verification and strategy-selection phases when the underlying model lacks grounded knowledge of how actions and environmental changes actually produce outcomes in the system?

Findings

Executive Summary

Adversarial inputs and environmental shifts propagate through multi-step tool-using Large Language Model systems mainly by corrupting the working state that later strategy and verification steps reuse, not by causing one isolated bad output.

When verification is implemented as another free-form judgment from the same model on the same context, it is usually not an independent check and cannot reliably detect blind-spot-consistent errors.

Prompt injection, meaning malicious instructions embedded in content that the model later treats as authoritative instructions, is the clearest empirical example in this item because attacker text can be ingested as data, reinterpreted as instructions, and then propagated through tool calls, memory summaries, or future plans.

The best-supported formal picture is an amplification process in which error grows whenever path dependence and persistence outpace independent correction, which means agent loops without external verifiers mostly redistribute and sometimes magnify causal ignorance rather than curing it.

Key Findings

Indirect prompt injection succeeds because tool-using Large Language Model applications routinely concatenate retrieved content with user instructions, leaving models unable to reliably separate informational context from actionable commands.
Once the loop's perception stage encodes a corrupted task state, strategy selection can remain internally coherent while optimizing for the wrong objective, so later tool calls inherit rather than repair the original semantic error.
Tool use and persistent memory can extend adversarial corruption beyond one response by letting a locally plausible mistake change external state and, in documented cases, influence later sessions.
Generic self-verification by the same model class is not an independent safety layer, because intrinsic self-correction remains weak without external feedback and usually inspects the same contaminated context that produced the original mistake.
Structured verification can recover some reasoning errors only when it introduces extra constraints such as explicit condition isolation, which shows that correction comes from added independence rather than from reflection alone.
Unexpected environmental shifts and adversarial prompting are propagation-equivalent at the loop level when they both create observations that look familiar to the model while no longer preserving the environment's real causal structure.
A useful formal approximation is that expected semantic error grows across repeated steps whenever path-dependent amplification and persistence exceed the error removed by independent correction signals.

Assumptions

The Anthropic agent-loop description and Research Question 4.1 together represent the common structure of contemporary multi-step tool-using systems closely enough for a structural propagation model.
Expected semantic error magnitude is an acceptable synthesis variable even though the cited studies report attack success, accuracy, or qualitative failure rather than one shared scalar measurement.
Environmental shift can be grouped with adversarial input for propagation analysis because the mechanism under study is misleading observation entering a recurrent loop, not attacker intent itself.

Analysis

The strongest evidence in this item is on indirect prompt injection and intrinsic self-correction, because those claims rest on direct primary studies and official security guidance rather than on analogy alone.

The weakest element is the cross-domain bridge from classical adversarial examples to language-model agent loops, so Goodfellow et al. is used here to motivate the adversarial-input framing rather than as direct evidence about tool-using Large Language Model applications.

Wu et al. is the main rival interpretation because it shows that models can sometimes improve answers during verification, but its gains require key-condition isolation rather than unconstrained reflection, which supports rather than contradicts the independence claim.

Another rival explanation is that failures arise only from overly permissive tools rather than from model-level causal limits, but Greshake, Yi, and Microsoft all show that misclassification of context as instructions already misroutes the loop before any one tool policy is discussed.

The formal propagation model is therefore best read as a structural synthesis: attack severity depends on path dependence, persistence, and verifier independence, not only on raw model accuracy at one isolated step.

Risks, Gaps, and Uncertainties

No cited source directly estimates one shared numeric amplification coefficient across full plan-act-verify loops, so the formal equation is structural rather than benchmark-calibrated.
The cited evidence on cross-session memory poisoning in this item comes from a proof of concept and vendor guidance rather than from a multi-platform benchmark study.
The environmental-shift argument is strongest at the structural level and weaker at the level of one shared empirical benchmark that jointly measures shift, tool use, and verification failure in the same loop.

Open Questions

Which benchmark design best measures full-loop propagation, including planning, tool use, memory, and verification, under both prompt injection and non-malicious environmental shift?
How much independent correction is added by deterministic policy engines, typed tool interfaces, or formal verifiers compared with model-only verification?
When does memory persistence improve robustness by preserving context, and when does it mainly extend the lifetime of semantically corrupted state?