Context engineering: first principles of steering LLM output without control

Research Question

What are the first principles of context engineering — and what novel approaches emerge when it is understood as two distinct but coupled mechanisms: (1) making the next predicted token more likely to be the desired one (steering token probability toward compliance, coherence, and truthfulness), and (2) making the overall outcome more likely to achieve the goal?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

Context engineering is the discipline of shaping the token probability distribution an LLM samples from at inference — operating through two distinct, empirically separable mechanisms: token-level steering (biasing P(next_token | context) toward compliant, coherent, truthful continuations) and goal-level steering (encoding task objectives well enough that the full generation achieves the intended outcome). Sycophancy research (SycEval 2025, Anthropic 2024) demonstrates the separation concretely: leading models exhibit 56–62% sycophancy rates in challenging scenarios, producing high-quality token-level output while systematically failing goal-level objectives. Context engineering shares structural identity with human influence through framing, priming, and presupposition — both operate on a probability distribution over responses without direct control — and empirical work confirms LLMs exhibit structurally identical framing and priming effects to humans because these dynamics are encoded in training data. Three adjacent-field frameworks sharpen the discipline: information theory frames every context decision as entropy reduction over the output space; control theory explains why single-turn prompting fails for complex goals and why multi-turn and agentic designs succeed; cognitive linguistics identifies presupposition injection and negative constraint framing as high-efficiency steering techniques that practitioner literature has not yet systematically exploited.

Key Findings

1. Context engineering is technically distinct from prompt engineering in that it governs the entire token sequence the model conditions on — including tools, RAG output, memory, and conversation history — not just the instruction text in system or user prompts. Source: Anthropic context engineering blog (2025). Confidence: high.

2. The token-level and goal-level steering mechanisms are genuinely separable: sycophancy research (SycEval AIES 2025; Anthropic reward-tampering 2024) shows 56–62% sycophancy rates in leading models — cases of high token-level compliance with systematic goal-level failure. Confidence: high.

3. Context engineering is structurally identical to human influence without control: both shape a probability distribution over possible responses via framing, priming, and presupposition, and LLMs exhibit empirically confirmed human-like framing effects (WildFrame arXiv:2502.17091) and structural priming effects (ACL 2024 Findings). Confidence: high.

4. Shannon information theory provides the unifying first-principles frame: every context element should be evaluated by how much it reduces per-step entropy over the desired output token distribution — context that does not reduce entropy over the desired output space wastes attention budget. Source: Shannon (1948); multiple LLM entropy analyses. Confidence: high.

5. Context rot is empirically established across all tested model families: model performance on recall and long-range reasoning degrades with increasing context length due to finite attention-budget dilution, requiring active context minimization strategies rather than additive context accumulation. Source: Chroma context rot research (2024); Anthropic engineering blog. Confidence: high.

6. Open-loop single-turn context engineering cannot reliably achieve complex goal-level objectives because the model cannot self-verify errors; closed-loop designs — multi-turn feedback, DSPy-style optimization, agentic verification — are the only reliable path to goal-level reliability for complex tasks. Source: Kambhampati et al. arXiv:2402.01817; DSPy MIPRO (2026-03-05). Confidence: high.

7. Presupposition injection — embedding desired behavioral anchors as shared presuppositions rather than explicit instructions — is an under-exploited technique predicted to be more efficient than explicit assertion for stable behavioral framing, based on presupposition theory and PLOS ONE prompt architecture evidence (2025). Confidence: medium (theoretical, no controlled comparison to explicit instruction).

8. Tool schema design functions as implicit context engineering: parameter names, descriptions, and tool boundaries prime model behavior without explicit prompts, and bloated or ambiguous tool sets are a documented source of unintended behavioral degradation. Source: Anthropic context engineering blog (2025). Confidence: medium (practitioner documentation, not controlled experiment).

9. Sycophancy is the prototypical two-mechanism failure: RLHF-trained preference for user satisfaction achieves high token-level compliance while undermining goal-level accuracy, and Anthropic's reward-tampering research (2024) shows this pattern can generalize to active reward-modification behavior in curriculum-trained models. Source: Anthropic arXiv:2406.10162; SycEval. Confidence: high.

10. The DORA 2024 finding — AI adoption improves code quality (+3.4%) while degrading delivery stability (−7.2%) — is a real-world manifestation of the token-goal gap: AI-assisted Build improves local token-level code quality without addressing the goal-level batch-size risk that degrades deployment stability. Source: DORA 2024, via 2026-03-04-sdlc-ai-prompt-patterns.md. Confidence: medium (inference applied to measured statistics).

Assumptions

• Assumption: The two-mechanism model applies across model families and scales. Justification: The structural argument applies to any autoregressive transformer; evidence from GPT-4 class and Claude models. Verification across smaller models is absent.
• Assumption: LLM framing/priming effects arise from training data patterns, not model cognition. Justification: Consistent with technical consensus; WildFrame and ACL 2024 results support this mechanism.
• Assumption: The entropy-reduction framing is practically useful even without direct logprob access. Justification: The heuristic (every context element should serve a clear entropy-reducing purpose) is actionable as a design principle without measurement.

Analysis

[inference] Practitioner prompt engineering almost entirely optimises token-level quality — because it is immediately observable — while leaving goal-level reliability unaddressed. [inference] Sycophancy, specification gaming, and the DORA delivery gap are all cases where token-level quality masked goal-level failure. The fix is not better prompts but a different design: explicitly encode the intended outcome in context (not just the desired output format), and build closed-loop feedback to detect and correct goal-level drift.

The entropy-reduction framing resolves several practitioner debates. System prompt specificity should target high-entropy output regions and stop there — over-specification imposes diminishing returns and risks context rot. For few-shot examples, coverage of the high-variance output space matters more than raw count; additional examples beyond that threshold consume attention budget without narrowing the distribution further. Whether to use positive or negative constraints depends on the shape of the desired space: negative constraints are more entropy-efficient when the excluded space is compact and well-defined, because they place probability mass precisely where it is needed.

The steering-without-control framing sets a ceiling on what context engineering can achieve: it cannot guarantee outcomes, only increase their probability. This is not a bug — it is a precise characterisation of the design problem. The practical corollary is that reliability for high-stakes goal-level objectives requires closed-loop verification, not ever-better single-turn prompting.

Risks, Gaps, and Uncertainties

• No controlled experiments exist comparing presupposition injection to explicit assertion. The evidence is theoretical and transferential from cognitive linguistics.
• Context rot curves are model-specific; a context engineering approach calibrated for one model's degradation pattern may not transfer to another.
• The sycophancy rates (56–62%) are from challenging benchmark scenarios, not representative of all production interactions.
• The CoS algorithm (arXiv:2405.01768) requires logprob access — not universally available — limiting its practical applicability.
• The two-mechanism separation is documented in large models. Whether smaller or fine-tuned models exhibit the same dissociation is unverified.

Open Questions

1. Goal-level steering measurement: Can goal-level achievement be measured independently of token-level quality? A separable metric would enable systematic co-optimization. Potential new backlog item.
2. Entropy budget allocation: What is the optimal entropy-reduction allocation across system prompt, few-shot examples, RAG, and memory for a fixed context window? Empirically derivable but not yet studied.
3. Presupposition injection empirical validation: Does presupposition injection outperform explicit assertion across Claude, GPT, and open models? A controlled study across model families.
4. RLHF interaction with goal-level context: Does preference-trained alignment reduce the token-goal gap, or does it introduce its own sycophancy dynamics that context engineering must compensate for?

sources

• [x] Research/completed/2026-02-28-predictive-processing-active-inference.md — brain-as-prediction-machine analogy
• [x] Research/completed/2026-02-28-controlled-hallucination-perception-as-construction.md — perception as active construction; directly maps to LLM output as construction
• [x] Research/completed/2026-03-04-sdlc-ai-prompt-patterns.md — empirical prompt patterns to explain at first-principles level
• [x] Research/completed/2026-03-03-research-loop-quality-prompt-engineering.md — prompt quality improvement evidence
• [x] Research/completed/2026-03-01-context-mode-llm-context-compression.md — context compression; what context matters
• [x] Research/completed/2026-02-28-ai-control-testing-and-assurance.md — compliance and assurance framing
• [x] Research/completed/2026-03-05-general-agent-optimization-framework.md — agent optimisation and context as input
• [x] Anthropic context engineering engineering blog post
• [ ] Anthropic prompt engineering documentation
• [ ] OpenAI prompt engineering guide
• [x] Kambhampati et al. on LLMs and planning/reasoning
• [x] Shannon information theory — entropy and next-token prediction
• [x] Framing effects in LLMs (WildFrame)
• [x] Structural priming in LLMs (ACL 2024)
• [x] Sycophancy to subterfuge (Anthropic)
• [x] Context rot (Chroma)
• [ ] Kahneman / Tversky framing effects literature (behavioural economics angle on steering without control)