Coverage gaps in automated research review skills, peer review patterns for…

Research Question

What review methodology is required to reliably move from information gathering through to applied knowledge and wisdom — and which of those steps can be automated in a CI pipeline versus requiring human or peer-level judgement?

Findings

Executive Summary

The three existing quality skills (citation-discipline, speculation-control, remove-ai-slop) cover factual hygiene and writing quality but leave three critical gaps: logical coherence (do conclusions follow from evidence?), alternative explanations (were competing interpretations considered?), and cross-item integration (does this item connect to the wider corpus?). A four-tier quality review pipeline, sequenced by automation cost, closes these gaps: Tier 1 structural checks (fully automatable Python scripts), Tier 2 single-item agent review (existing skills plus new logical coherence and evidence sufficiency checks), Tier 3 cross-item integration checks (agent with corpus access), and Tier 4 domain-specific human review. A new peer-review skill for davidamitchell/Skills is warranted to formalise the logical coherence and evidence sufficiency checks; an integration skill is not yet warranted. The DIKW and SECI frameworks confirm that the Wisdom level requires cross-item synthesis (Combination) and application to real decisions (Internalisation) — steps that the existing individual-item review pipeline does not address and which are covered by the synthesis and knowledge-retention work already designed in this corpus.

Key Findings

1. [fact] The three existing skills leave logical coherence, alternative explanations, and cross-item integration uncovered. Citation-discipline checks source presence; speculation-control checks label presence; remove-ai-slop checks prose surface patterns — none checks whether a conclusion follows from the evidence, whether competing interpretations were considered, or whether the item connects to related completed items in the corpus.

2. [fact] LLM agents can check logical coherence and missing alternative explanations at a quality level matching median human peer reviewers. REMOR (arXiv:2505.11718) and DeepReview (ACL 2025) demonstrate this on scientific manuscripts; the Copilot CLI agent already demonstrates equivalent behaviour in research-review.yml. The boundary between agent-automatable and human-only is coherence vs. domain plausibility — not "agent vs. human."

3. [inference] A four-tier quality review pipeline sequenced by automation cost is the correct architecture: structural checks → single-item agent review → cross-item agent review → human domain review. Each tier gates the next; cheaper checks run first to fail fast before running expensive LLM calls or requiring human attention.

4. [inference] The existing research-review.yml Tier 2 agent review should be extended with two new checks: logical coherence (does §6 Synthesis follow from §2 Investigation?) and evidence sufficiency (is the confidence level calibrated to source count and independence?). These are the highest-impact missing checks — they catch the failure mode where a well-sourced item misinterprets its own evidence and reaches an incorrect conclusion.

5. [fact + inference] The DIKW and SECI frameworks map to three distinct workflow stages in this repository: Information→Knowledge (individual item research and review), Knowledge→Wisdom (cross-item synthesis via the planned synthesise.yml), and Wisdom application (internalisation via the knowledge-retention mechanisms). Individual item quality review only covers the first transition; the second and third require separate workflows that are designed but not yet built.

6. [inference] Cross-item integration is a quality dimension for individual items, not only a synthesis-layer concern. A research item that references no related completed items and creates no links in state/links.json is a quality failure — the Zettelkasten principle (connection generates insight) means isolated items accumulate information without advancing knowledge.

7. [inference] A peer-review skill with three checks is the correct scope addition to davidamitchell/Skills. The three checks: (1) the Executive Summary conclusion is supported by §2 Investigation evidence; (2) at least one major alternative explanation was considered or explicitly excluded; (3) confidence levels are calibrated to source count and independence per the research skill's confidence table.

8. [fact] Bloom's revised taxonomy (2001) maps directly onto the four automation tiers: Remember/Understand levels map to structural checks; Apply/Analyse levels map to agent reasoning; Evaluate/Create levels map to human judgment. This is not a loose analogy — cognitive complexity at each Bloom level correlates with the computational cost and contextual requirements of the corresponding automation tier.

9. [inference] The human review tier covers only domain plausibility and strategic relevance, not logical validity. In the NZ financial services context, acting on a finding without domain plausibility review carries RBNZ supervisory risk; this makes Tier 4 non-optional for decisions with compliance implications, even though it is narrow in scope.

10. [inference] An integration skill is not yet warranted; it becomes necessary once Research/synthesis/ documents exist and require quality review of their own. The peer-review skill's cross-item reference check and the existing synthesis workflow design are sufficient for current corpus size and state.

Assumptions

• Assumption: The Copilot CLI agent running research-review.yml has sufficient reasoning capability to perform logical coherence checks on research items. Justification: REMOR and DeepReview demonstrate that current LLM-class models match median human reviewer quality on coherence; the Copilot CLI uses Claude Sonnet, which is in the same capability class. Direct empirical validation in this specific repo would be needed to confirm.
• Assumption: The existing research-review.yml agent prompt can be extended with logical coherence and evidence sufficiency instructions without degrading the quality of the existing three skill checks. Justification: The existing prompt already chains three separate skill checks; adding two more checks in the same format is consistent with the established pattern. The risk is prompt length degradation, but the existing prompt is already ~120 lines and functions correctly.
• Assumption: "Domain plausibility" checks (Tier 4) are not LLM-automatable for NZ financial services context. Justification: The ICLR 2025 feedback confirms agents underperform on domain-specific novelty and plausibility; the RBNZ supervisory expectations item confirms that NZ-specific regulatory context is required for correct domain assessment. This is a strong inference given the evidence, not an empirically proven claim about this specific agent.

Analysis

The skills gap analysis is the foundational finding: all subsequent design choices follow from identifying exactly what the existing three skills check and what they miss. The peer review literature (Springer Nature, REMOR, DeepReview) provides external validation that the identified gaps — logical coherence and alternative explanations — are the same dimensions academic peer review prioritises. This convergence across independent frameworks (skill analysis + academic peer review + LLM research) gives high confidence in the gap identification.

The DIKW/SECI mapping resolves a potential confusion: "quality review" and "knowledge integration" are often conflated, but they operate at different pipeline stages. Quality review is an individual-item concern; knowledge integration is a cross-item concern; wisdom requires application beyond the repository. The four-tier pipeline reflects this: Tiers 1–2 address individual item quality; Tier 3 addresses integration; Tier 4 addresses domain-specific applicability. The tiers are not just ordered by cost but by epistemic depth — each tier catches failures the previous cannot.

The decision not to create an integration skill reflects scope discipline: the synthesis workflow already covers Combination, and the peer-review skill's cross-reference check is sufficient for Tier 3. Adding a third new skill without a concrete failing case would be premature abstraction.

Risks, Gaps, and Uncertainties

• Ahrens (2017) was not accessed (book inaccessible). The Zettelkasten findings are corroborated by zettelkasten.de primary documentation and two prior completed items; this gap does not affect any Key Finding.
• Empirical validation of LLM coherence checking in this specific repo has not been done. The claim that the Copilot CLI agent can perform logical coherence checks is supported by REMOR/DeepReview but not validated against this repo's items. The first run of the extended research-review-prompt.md will provide this validation.
• Tier 3 cross-item check implementation complexity is uncertain. Requiring the agent to list Research/completed/ and compare tags requires either filesystem access in the CI context (currently available via actions/checkout) or a pre-built tag index. The detail design is left to BACKLOG.md W-0031.
• The peer-review skill scope is the right starting point but may require iteration once used in practice. The three checks (conclusion-evidence support, alternative explanations, confidence calibration) are defensible but a first run may surface additional edge cases.

Open Questions

• Should Tier 1 structural checks block the research loop commit itself, or only gate the post-commit review workflow? Blocking the loop commit on structural failures would catch malformed items earlier but would require integrating structural checks into research-loop.yml, increasing its complexity.
• What is the correct trigger for Tier 4 human review? CI cannot enforce human review, but a GitHub issue template for "domain plausibility review needed" could make it visible. Whether this should be automatically created for every completed item or only for items tagged with specific domains (e.g., rbnz, financial-services) is a design choice for BACKLOG.md W-0031.
• When does an integration skill become warranted? The threshold is: when Research/synthesis/ documents exist and need their own quality review. This should be tracked as a condition in the synthesis workflow backlog item.
• Can Tier 4 human review be partially automated for the NZ context by injecting the RBNZ supervisory expectations findings and the AI strategy completed items as context into the agent prompt? This would make domain plausibility partially agent-automatable for the documented NZ context, though it would still require owner validation for novel decisions.

Output section

• Type: skill, knowledge, backlog-item
• Description: Four-tier quality review pipeline taxonomy with automation classification; peer-review skill scope; DIKW/SECI mapping to repository workflows. The direct input to BACKLOG.md W-0031 (research review CI step implementation).
• Links:
  • Academic peer review process documentation — Springer Nature peer review guidelines (Springer Nature peer review guidelines — defines what reviewers check)
  • arxiv.org (REMOR — LLM peer review matching human quality on coherence)
  • Zettelkasten method (Zettelkasten introduction — integration methodology foundation)

sources

• [x] Ackoff (1989) — "From Data to Wisdom" — foundational DIKW hierarchy (secondary sources consulted; primary DOI paywalled)
• [x] Nonaka & Takeuchi (1995) — The Knowledge-Creating Company — SECI model (secondary sources consulted)
• [x] Bloom's Taxonomy (1956, revised 2001) — cognitive levels as a framework for knowledge depth vs applicability
• [x] Zettelkasten method — atomic notes, linking, emergence of insight through connection
• [ ] Ahrens (2017) — How to Take Smart Notes — practical Zettelkasten for researchers; integration as a first-class practice (book not accessible)
• [ ] GitHub Actions documentation — composite actions, reusable workflows — for CI pipeline design (not consulted; sufficient knowledge from prior research)
• [x] Existing skills: .github/skills/citation-discipline/SKILL.md, .github/skills/speculation-control/SKILL.md, .github/skills/remove-ai-slop/SKILL.md — gap analysis starting point
• [x] Academic peer review process documentation — Springer Nature peer review guidelines
• [x] LLM automated peer review — REMOR (arXiv:2505.11718), DeepReview (ACL 2025), LLM4Review (OpenReview)
• [x] Related completed items: 2026-03-03-cross-item-synthesis-meta-insights.md, 2026-03-03-research-loop-quality-prompt-engineering.md, 2026-03-03-knowledge-retention-active-recall.md, research-review-prompt.md, .github/workflows/research-review.yml