Research / Evaluating and improving autonomous research loop quality

Evaluating and improving autonomous research loop quality

Evaluating and improving autonomous research loop quality: prompt engineering and output assessment

2026-03-03 · agentic-ai benchmarks-eval tools-infrastructure llm-reasoning · medium · source → · wiki →
key claims
  1. Items with the Research Skill Output section (§0–§7) produce dramatically better outputs: avg 12.8 sources checked vs. 3.1 without it, and avg 43 key findings vs. 14. The section is the most impactful quality lever already deployed
  2. 40% of pre-2026-03-03 items have at least some sources listed but not consulted; 20% (4/20) have all sources unchecked. This is the primary source engagement failure and is prompt-caused: no explicit instruction existed to mark sources `` when consulted
  3. The current `research-prompt.md` lacks an instruction to search `Research/completed/` for prior related work before starting a new item. As the corpus grows beyond 20 items this gap becomes a structural deficiency: each new item that ignores prior completed work duplicates effort and misses synthesis opportunities
  4. Priority ordering in the loop is correct and respected in practice: `research-prompt.md` contains explicit `priority: high → medium → low` rules, confirmed by both document inspection and observed execution order. No fix required
  5. Chain-of-thought prompting in the form of structured step decomposition (§1 → §2 → §6) is the mechanism behind the Research Skill Output section's quality improvement, consistent with The Prompt Report (arXiv:2406.06608) documenting CoT as one of 58+ validated techniques
  6. Negative constraints work better when paired with positive instructions. "Do not reproduce template headings without substantive content" should be paired with "each section must contain defensible prose before the next heading."
  7. The Fabric `extract_wisdom` pattern's use of minimum output counts per section is a more reliable specificity enforcement mechanism than negative constraints alone. Adapting this to Key Findings — requiring each finding to be ≥ 20 words — is the direct route to eliminating one-line vague findings
  8. A four-check shell quality gate detects the three primary failure modes at low false-positive cost: completed date populated, executive summary word count ≥ 80, ≥ 2 key findings with ≥ 15 words each, and ≥ 1 source marked ``

Research Question

How can the quality of research items produced by the research-loop.yml autonomous pipeline be systematically evaluated, and what changes to research-prompt.md and the loop's prompting strategy would produce higher-quality, deeper, more synthesis-rich research outputs?

Findings

Executive Summary

The quality variation in autonomous research loop output is prompt-caused, not model-caused: the model is capable of high-quality synthesis, and corpus evidence demonstrates this directly. The Research Skill Output section (§0–§7) added to research-prompt.md increased source engagement from 3.1 to 12.8 sources per item and key finding counts from 14 to 43, making it the single most impactful quality change to date. Three residual failure patterns persist in the pre-2026-03-03 corpus: sources listed but not consulted (40% of items), executive summaries that lack specific claims, and no cross-item integration with prior completed research — all three are addressable by five targeted additions to research-prompt.md. A four-check shell quality gate (completed date, ≥80-word executive summary, ≥2 substantive findings, ≥1 checked source) detects the most severe failures and can be implemented as a non-blocking GitHub Actions annotation step.

Key Findings

  1. Items with the Research Skill Output section (§0–§7) produce dramatically better outputs: avg 12.8 sources checked vs. 3.1 without it, and avg 43 key findings vs. 14. The section is the most impactful quality lever already deployed. [high confidence]

  2. 40% of pre-2026-03-03 items have at least some sources listed but not consulted; 20% (4/20) have all sources unchecked. This is the primary source engagement failure and is prompt-caused: no explicit instruction existed to mark sources [x] when consulted. [high confidence]

  3. The current research-prompt.md lacks an instruction to search Research/completed/ for prior related work before starting a new item. As the corpus grows beyond 20 items this gap becomes a structural deficiency: each new item that ignores prior completed work duplicates effort and misses synthesis opportunities. [high confidence]

  4. Priority ordering in the loop is correct and respected in practice: research-prompt.md contains explicit priority: high → medium → low rules, confirmed by both document inspection and observed execution order. No fix required. [high confidence]

  5. Chain-of-thought prompting in the form of structured step decomposition (§1 → §2 → §6) is the mechanism behind the Research Skill Output section's quality improvement, consistent with The Prompt Report (arXiv:2406.06608) documenting CoT as one of 58+ validated techniques. [high confidence]

  6. Negative constraints work better when paired with positive instructions. "Do not reproduce template headings without substantive content" should be paired with "each section must contain defensible prose before the next heading." [medium confidence]

  7. The Fabric extract_wisdom pattern's use of minimum output counts per section is a more reliable specificity enforcement mechanism than negative constraints alone. Adapting this to Key Findings — requiring each finding to be ≥ 20 words — is the direct route to eliminating one-line vague findings. [medium confidence]

  8. A four-check shell quality gate detects the three primary failure modes at low false-positive cost: completed date populated, executive summary word count ≥ 80, ≥ 2 key findings with ≥ 15 words each, and ≥ 1 source marked [x]. [high confidence]

  9. The "Lost in the Middle" effect (arXiv:2307.03172) means critical instructions should be placed near the top of the relevant step in research-prompt.md. The source-marking discipline instruction (addition 2) and prior-research cross-reference instruction (addition 1) must each appear as the first instruction in their respective steps to maximise compliance. [medium confidence]

  10. Fifteen items in Research/completed/ (pre-2026-03-03) have suboptimal source coverage due to the pre-Research Skill Output prompt. A targeted re-enrichment pass on the strategically most important items is a separate task worth adding to the backlog. [medium confidence]

Assumptions

Analysis

The corpus audit produces a clean before/after split at 2026-03-03. The Research Skill Output section addition was decisive; the proposed five additions are incremental refinements targeting the residual gaps it does not fully close. The most impactful addition is the prior research cross-reference instruction: it transforms the corpus from a collection of isolated items into a connected knowledge base, compounding in value as item count grows.

Prompt length management is straightforward: from ~800 to ~1,100 words, within the range where instruction-following remains reliable. The critical placement guidance (instructions near the top of their steps) mitigates the "Lost in the Middle" risk without restructuring the prompt.

The quality gate is designed as an informational signal initially. This avoids the risk of a conservative threshold blocking a legitimate item and stalling the loop. After calibration (confirming zero false positives across 5+ loop runs), it can be made blocking.

Risks, Gaps, and Uncertainties

Open Questions

sources

← newerLocal database: requirements and technology choice
older →Research agenda curation

Connected items

Loading…

View full knowledge graph →