Closed-loop performance techniques for mediated execution harnesses

2026-08-20 · agentic-ai memory-context cost-performance benchmarks-eval tools-infrastructure · medium · source → · wiki →

key claims

A controlled comparison inside the SWE-agent scaffold against the SWE-bench Verified benchmark found that observation masking, simply hiding older raw tool outputs, matched or exceeded the task solve rate of LLM-generated summarisation while roughly halving token cost across five model configurationsLindenbauer (2025)Research (2025)
Model recall degrades non-uniformly as input length increases even when task difficulty is held constant, evidenced across an 18-model evaluation that intentionally isolated input length from task complexityResearch (2025)
Large language models retrieve information placed in the middle of a long context substantially worse than information placed at the beginning or end, a positional degradation pattern distinct from and additive to raw length effectsArxiv (n.d.)
Retrieval accuracy collapses far more steeply with context length when the match between a query and the needed information is semantic rather than lexical, showing that Needle in a Haystack (NIAH)-style lexical benchmarks overstate real long-context capabilityModarressi (2025)
A verbal self-reflection loop that stores natural-language feedback about a specific failed attempt and consults it on the next attempt raised HumanEval pass@1 accuracy from a baseline in the low-to-mid 80s percent to 91% without any update to model weightsShinn (2023)
Both major symbolic cognitive architectures, Adaptive Control of Thought-Rational (ACT-R) and Soar, separate a small bounded working memory that holds only currently active information from an unbounded long-term procedural store that is updated incrementally from problem-solving outcomes, a structural pattern that MemGPT's fast/slow memory tiers and Reflexion's episodic-reflection store independently reproduce in software agentsJones (2022)Packer, Fang, et al. (2023) MemGPT (2023)Shinn (2023)
Denning's working-set model defines a system as thrashing, meaning throughput collapses, once the sum of active processes' referenced-page sets exceeds available memory, giving a directly adaptable definition for a harness "working-set utilisation" metric as the fraction of active context budget occupied by content the model actually references for its next decisionDenning (1968)
Feedback-latency thresholds established for human perception of a directly operated interface (roughly instantaneous below 0.1 seconds, uninterrupted flow up to 1 second, attention loss beyond 10 seconds without explicit progress feedback) have not been directly measured against harness-internal tool-call or diagnostic latency in any source reviewed for this itemNielsen (1993)

Research Question

What measurable techniques from closed-loop control, working-memory management, adaptive procedural systems, and real-time feedback design improve the performance of an active execution harness that consumes a fixed mediation layer, and which indicators best demonstrate that those techniques remain effective across different underlying content pools?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

The single best-supported harness-performance lever identified in this investigation is discarding or masking stale context rather than compressing it through summarisation, because a controlled comparison on the Software Engineering Benchmark (SWE-bench) Verified dataset found observation masking cheaper and at least as effective as large language model (LLM) summarisation across multiple model configurations. The evidence that effective context capacity is smaller than nominal context-window size is strong and independently replicated across at least three separate evaluations spanning 18 models, positional recall tests, and semantic-match retrieval tests. Execution-feedback loops that let a harness verbally reflect on a specific failed attempt and consult that reflection on the next attempt produce a measured accuracy gain (80% to 91% pass@1 on HumanEval) without any change to model weights or the content pool. Classical control-theory stability metrics such as gain margin and phase margin have no demonstrated harness-level measurement in the literature surveyed and remain a structural analogy rather than a transferable indicator today. Four cross-pool-transferable indicator families emerge from the evidence: token-normalised working-set utilisation, positional/semantic recall degradation rate, feedback-attribution granularity, and cost-adjusted recovery rate, none of which were demonstrated as a single unified benchmark in any one source.

Key Findings

A controlled comparison inside the SWE-agent scaffold against the SWE-bench Verified benchmark found that observation masking, simply hiding older raw tool outputs, matched or exceeded the task solve rate of LLM-generated summarisation while roughly halving token cost across five model configurations.
Model recall degrades non-uniformly as input length increases even when task difficulty is held constant, evidenced across an 18-model evaluation that intentionally isolated input length from task complexity.
Large language models retrieve information placed in the middle of a long context substantially worse than information placed at the beginning or end, a positional degradation pattern distinct from and additive to raw length effects.
Retrieval accuracy collapses far more steeply with context length when the match between a query and the needed information is semantic rather than lexical, showing that Needle in a Haystack (NIAH)-style lexical benchmarks overstate real long-context capability.
A verbal self-reflection loop that stores natural-language feedback about a specific failed attempt and consults it on the next attempt raised HumanEval pass@1 accuracy from a baseline in the low-to-mid 80s percent to 91% without any update to model weights.
Both major symbolic cognitive architectures, Adaptive Control of Thought-Rational (ACT-R) and Soar, separate a small bounded working memory that holds only currently active information from an unbounded long-term procedural store that is updated incrementally from problem-solving outcomes, a structural pattern that MemGPT's fast/slow memory tiers and Reflexion's episodic-reflection store independently reproduce in software agents.
Denning's working-set model defines a system as thrashing, meaning throughput collapses, once the sum of active processes' referenced-page sets exceeds available memory, giving a directly adaptable definition for a harness "working-set utilisation" metric as the fraction of active context budget occupied by content the model actually references for its next decision.
Feedback-latency thresholds established for human perception of a directly operated interface (roughly instantaneous below 0.1 seconds, uninterrupted flow up to 1 second, attention loss beyond 10 seconds without explicit progress feedback) have not been directly measured against harness-internal tool-call or diagnostic latency in any source reviewed for this item.
The Observe-Orient-Decide-Act (OODA) decision-cycle model implies that reducing a harness's cycle time, independent of the accuracy of any single cycle, is itself a distinct performance lever, reframing "adaptation rate" as cycle frequency rather than per-cycle correctness.
Industry-consensus software-delivery metrics measure failed-change recovery time as a validated throughput/stability indicator, corroborating "error-recovery speed" as a transferable, already-standardised measurement category outside the AI-agent literature.
Cognitive-architecture adaptation (Soar's chunking, Reflexion's per-attempt reflection) is tied to attribution granularity, learning a specific rule from a specific subgoal or failed attempt rather than treating an entire episode as one undifferentiated signal, which suggests harness adaptation rate should be measured jointly with feedback specificity rather than cycle frequency alone.

Assumptions

Classical control-theory stability metrics (gain margin, phase margin) are assumed to have a conceptual harness-level equivalent even though no cited source measures them on an AI agent harness. The justification is that feedback latency's effect on phase margin is well established in general control engineering, and the sensor/setpoint/actuator structure maps cleanly onto harness components, even without a direct harness-level measurement.

Nielsen's human-computer interaction response-time thresholds are assumed to be a reasonable starting reference for harness-internal feedback latency even though they were derived for direct human perception rather than agent-internal signalling. Both settings serve the same underlying purpose, keeping an actor's model of task state from going stale, which supports carrying the thresholds over even though the actor differs, a human user versus a harness's own control loop.

The Denning working-set threshold is assumed to be directly adaptable to a harness's context budget without modification, even though the original model was built for discrete memory pages rather than continuous token streams. Both systems share the same defining property, a bounded active resource whose useful fraction can be measured against a recent reference window, which supports the transfer despite tokens and pages differing in granularity and cost structure.

Analysis

The strongest evidence in this item clusters around context-window management because it is the only sub-question with a direct, controlled, multi-model comparison of competing techniques (masking versus summarisation versus no management) rather than a single demonstration or an architectural analogy. Weighing this against the MemGPT paging architecture required distinguishing what each source actually evaluated: Lindenbauer et al. tested single-session coding tasks where stale tool outputs lose value quickly, while Packer et al. tested document analysis and multi-session chat where information must remain recoverable indefinitely, so the two findings support complementary rather than competing recommendations depending on whether a harness's task class tolerates losing stale context outright. Reflexion's execution-feedback finding was weighed as strong because it is a controlled ablation with a specific mechanism isolated (the reflection step) and a measured before/after benchmark score, rather than a correlational or anecdotal claim. The control-theory and OODA mappings were deliberately weighed as the weakest tier of evidence in this item, retained as structural analogies because they supply useful vocabulary and hypotheses (settling time as convergence iterations, cycle time as a distinct lever from per-cycle accuracy) but downgraded to inference or assumption throughout because no cited source measures them directly on an AI execution harness. Rival explanations for the observation-masking result were considered: it is possible that masking's advantage is specific to SWE-bench Verified's task structure rather than a general property, but the JetBrains summary reports the effect generalising from the SWE-agent scaffold to the OpenHands scaffold, which weighs against a narrow, single-scaffold explanation without fully ruling out a benchmark-specific effect.

Risks, Gaps, and Uncertainties

No source consulted for this item directly instruments an AI coding or research harness with control-theory-style stability metrics (gain margin, phase margin, settling time in a formally measured sense); the mapping in this item is structural and unverified by direct measurement, and a targeted search for such a study found none.

search_query: "gain margin" OR "phase margin" AI agent harness benchmark evaluation
outcome: no matching paper or benchmark located

No source directly compares the observation-masking/summarisation result against the MemGPT paging architecture on a shared benchmark, so which approach generalises further across content-pool types (bounded coding sessions versus unbounded document/conversation analysis) remains an open empirical gap rather than a resolved finding.

Nielsen's response-time thresholds and Boyd's OODA tempo model were both developed outside the AI-agent domain decades before LLM-based harnesses existed, and this item's mapping of them onto harness internal loops has not itself been empirically validated against harness telemetry; readers should treat findings 8 and 9 as hypotheses for future measurement rather than established indicators.

The item relies on a single controlled study (Lindenbauer et al.) for its strongest, highest-confidence claim about context-management technique comparison; while that study spans five model configurations and two scaffolds, it is still one research group's benchmark design, and independent replication by a different team has not been located.

This item treats error-recovery speed as a cost/latency metric without addressing the governance question of when a harness should be trusted to recover autonomously versus escalate to a human; the repository's prior item on agent process reliability architecture found that inference from behavioural traces should be limited to suggestion and exception handling rather than unreviewed execution authority, which implies a fast harness-internal recovery loop still needs an escalation boundary this item does not itself define.

Open Questions

What would a harness-level instrumentation of gain margin, phase margin, or settling time actually look like in practice, and could such a metric be constructed and validated against existing agent benchmarks?
Does the observation-masking advantage over summarisation hold on task families outside software engineering, for example long-document research or multi-step web navigation?
Can Denning's working-set utilisation metric be operationalised and measured directly inside an existing harness (for example this repository's own Copilot CLI sessions) rather than only proposed by analogy?
Does feedback-attribution granularity (Soar's chunking, Reflexion's per-attempt reflection) predict adaptation-rate differences better than raw cycle time (the OODA framing), and could both be measured on the same benchmark to test which matters more?