What systematic review methodologies and Artificial Intelligence (AI)-assisted…

What systematic review methodologies and Artificial Intelligence (AI)-assisted synthesis tool architectures are most appropriate for cross-item synthesis of a growing file-based research corpus, and what design prevents hallucination and claim conflation across source items?

2026-05-02 · ai-architecture knowledge-management benchmarks-eval tools-infrastructure rag-retrieval agentic-ai · medium · source → · wiki →

key claims

A defensible synthesis workflow for this corpus should combine Cochrane-style protocol discipline with narrative synthesis as the default integration method, because PRISMA improves transparency while narrative synthesis is the best fit for heterogeneous, non-meta-analytic evidencePage et al. (2021)McKenzie et al. (2024)McKenzie et al. (2024)Brennan (2024)Popay et al. (2006)
Realist synthesis and meta-ethnography are better treated as optional interpretive passes than as the base workflow, because they preserve context and mechanism but are narrower and more specialized than the repository's general cross-item synthesis needWong et al. (2013)France et al. (2019)Popay et al. (2006)
The repository's main anti-hallucination control should be claim extraction before prose generation, because retrieval grounding alone does not stop a Large Language Model (LLM) from conflating adjacent source-item claims into an unsupported synthesis sentenceRani et al. (2024)Li et al. (2024)Mitchell (2026)
Each synthesis claim needs a minimum provenance record that includes source item slug, source location, epistemic label, confidence rationale, and contradiction status, because transparent non-meta-analytic synthesis still depends on visible evidence mapping and grouping decisionsMcKenzie et al. (2024)Brennan (2024)Mitchell (2026)
STORM and LlamaIndex provide the strongest architectural patterns for this use case, because STORM broadens evidence collection through perspective-guided questioning while LlamaIndex preserves source nodes through iterative synthesisShao et al. (2024)Stanford (n.d.)LlamaIndex (n.d.)LlamaIndex (n.d.)
Generic map-reduce summarization is useful for scale but insufficient as the provenance layer, because it summarizes documents independently and then reduces summaries, which preserves throughput better than claim-level traceabilityLangChain (n.d.)LlamaIndex (n.d.)
Artificial Intelligence (AI)-assisted review tools such as Elicit should be treated as complementary accelerators rather than authoritative synthesizers, because empirical evaluation shows value in search and organization but also substantial variability and incomplete overlap with traditional review resultsTannou et al. (2025)Reviews (2024)
Contradictions must be first-class synthesis outputs rather than silent prompt-internal reasoning, because preserving context-dependent differences is essential to prevent false consensus and because prior repository work already defines usable cross-item relationship typesWong et al. (2013)France et al. (2019)Mitchell (2026)Mitchell (2026)

Research Question

What systematic review methodologies, Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA), Cochrane review, narrative synthesis, meta-ethnography, and realist synthesis, and what Artificial Intelligence (AI)-assisted knowledge synthesis tool architectures are most appropriate for producing accurate, provenance-preserving cross-item synthesis from a growing file-based research corpus of about 200 items managed by AI agents? More specifically: what synthesis methodology best prevents hallucination and claim conflation across source items, what provenance-linking mechanism ensures each synthesis claim traces to specific source items, and what workflow design, GitHub Actions workflow_dispatch, agent prompt, and output directory structure, best delivers a synthesis-prompt.md and synthesis-loop.yml implementation for W-0051?

Findings

Executive Summary

A systematic-review-inspired hybrid is the best design for this repository: Cochrane and PRISMA provide rigor, narrative synthesis provides the default method for integrating heterogeneous studies, and realist synthesis or meta-ethnography should be used only when the question is explicitly about mechanisms, contexts, or meanings.

Preventing hallucination in this corpus depends on extracting and clustering claims before prose generation, because retrieval grounding alone does not stop claim conflation across related source items.

Architecturally, the strongest pattern is STORM-style perspective expansion combined with LlamaIndex-style source-node retention, while map-reduce summarization should remain a bounded reduction tactic rather than the provenance layer.

Prior repository work on the exploration-synthesis gap also supports explicit agent-mediated synthesis, because exploratory work performed by agents is not always fully reconstructible by human supervisors.

Accordingly, the first implementation should remain manual-only, require explicit source_items and synthesis_question, write to Knowledge/, and carry an ADR for the new knowledge schema, provenance format, and publication-path changes.

Key Findings

A defensible synthesis workflow for this corpus should combine Cochrane-style protocol discipline with narrative synthesis as the default integration method, because PRISMA improves transparency while narrative synthesis is the best fit for heterogeneous, non-meta-analytic evidence.
Realist synthesis and meta-ethnography are better treated as optional interpretive passes than as the base workflow, because they preserve context and mechanism but are narrower and more specialized than the repository's general cross-item synthesis need.
The repository's main anti-hallucination control should be claim extraction before prose generation, because retrieval grounding alone does not stop a Large Language Model (LLM) from conflating adjacent source-item claims into an unsupported synthesis sentence.
Each synthesis claim needs a minimum provenance record that includes source item slug, source location, epistemic label, confidence rationale, and contradiction status, because transparent non-meta-analytic synthesis still depends on visible evidence mapping and grouping decisions.
STORM and LlamaIndex provide the strongest architectural patterns for this use case, because STORM broadens evidence collection through perspective-guided questioning while LlamaIndex preserves source nodes through iterative synthesis.
Generic map-reduce summarization is useful for scale but insufficient as the provenance layer, because it summarizes documents independently and then reduces summaries, which preserves throughput better than claim-level traceability.
Artificial Intelligence (AI)-assisted review tools such as Elicit should be treated as complementary accelerators rather than authoritative synthesizers, because empirical evaluation shows value in search and organization but also substantial variability and incomplete overlap with traditional review results.
Contradictions must be first-class synthesis outputs rather than silent prompt-internal reasoning, because preserving context-dependent differences is essential to prevent false consensus and because prior repository work already defines usable cross-item relationship types.
The first repository implementation should be a manual-only synthesis-loop.yml that requires explicit source_items and synthesis_question, because owner-selected scope is both the current repository norm and the safest control against low-quality bulk synthesis.
An ADR is warranted before W-0051 implementation because introducing Knowledge/, a new synthesis schema, and new site-rendering behavior changes the repository's information architecture and publication path, not just one workflow file.
Agent-mediated synthesis deserves explicit support in the design because prior repository research shows that exploratory work performed by agents is often not fully reconstructible by human supervisors, which weakens human-only synthesis as a transfer mechanism.

Assumptions

The first synthesis workflow will operate on owner-selected clusters rather than automatic whole-corpus batches, because that is the implementation pattern already fixed by W-0051 and reinforced by the existing research-loop safety model.
Existing completed items provide enough prior-art context to shape the provenance model without first implementing a full semantic-search or graph-database layer.

Analysis

The evidence does not support copying one named academic review method unchanged into the repository.

Instead, it supports separating three layers that are often muddled together in casual design discussions: review discipline, synthesis method, and generation architecture.

Review discipline comes from Cochrane and PRISMA, which force explicit boundaries, grouping logic, and visible reporting.

Default synthesis method comes from narrative synthesis, because the repository corpus is heterogeneous and design-oriented, while realist-synthesis and meta-ethnography are better invoked as specialized passes when the question is explanatory or interpretive.

Generation architecture should be grounded in perspective expansion, source-node retention, and visible contradiction handling, because those are the controls that directly counter false consensus and citation drift.

That architectural recommendation also aligns with prior repository research on the exploration-synthesis gap, which argues that agent-mediated synthesis becomes more necessary when the exploratory work itself is performed by agents and cannot be fully re-explained by humans afterward.

Risks, Gaps, and Uncertainties

The tool-comparison evidence is uneven, because open architectures such as STORM and LlamaIndex expose more inspectable detail than vendor-managed products.
No external benchmark in this evidence set directly measures claim conflation across repository-style Markdown items, so the recommended control stack remains evidence-informed design rather than benchmark-proven doctrine.
The recommendation to keep the first workflow manual-only is strongly justified by repository governance and safety constraints, but it is not a general property of systematic-review methodology.

Open Questions

Should the first Knowledge/ schema require excerpt-level evidence quotes, or is source item slug plus section identifier sufficient for version 1?
Should contradiction handling be stored only in the synthesis file, or also in machine-readable sidecar data for later tooling?
When the corpus grows further, should source-item clustering remain manual-only, or should a later version add opt-in semantic expansion after the provenance model is stable?