What technical architecture best supports cross-item synthesis, knowledge…

Research Question

What technical architecture best supports three distinct but related capabilities in a file-based research corpus (~200 items, growing weekly): (1) a meta-distillation layer that proactively aggregates findings across items to surface higher-order themes and emergent insights not visible in any single item; (2) a knowledge map that renders relationships between items, concepts, and tags as a connected, navigable structure that regenerates automatically; and (3) a search and synthesis interface that answers ad-hoc queries with provenance links and publishes new distilled insights without requiring manual intervention - and what are the concrete tool choices, index formats, active/reactive trigger designs, and Large Language Model (LLM)-vs-heuristic trade-offs for each?

Findings

Executive Summary

The best-fit architecture is a hybrid, file-based stack that refreshes deterministic graph and summary artifacts on every corpus change, then applies Large Language Model (LLM) synthesis only to shortlisted clusters or queries rather than to the whole corpus each time.

For the knowledge map, a precomputed nodes-and-edges JSON file rendered as a static D3 force graph is the strongest default for this repository's current corpus and publishing model, while Mermaid should be reserved for small thematic subgraphs and JSON-LD should be emitted as an interoperability export rather than as the primary runtime index.

For active insight generation, the lowest-risk trigger pattern is push-driven index refresh plus scheduled weekly distillation and on-demand workflow_dispatch query synthesis, because push keeps artifacts fresh, schedule amortizes expensive multi-item reasoning, and manual dispatch preserves a website-only control surface for ad hoc questions.

The retrieval layer should follow a Letta-style memory split: distilled per-item summaries, cluster manifests, and recent digests stay in prompt-resident working context, while the full Markdown corpus and graph artifacts remain searchable archival state on disk, which keeps provenance explicit without requiring a persistent database.

Key Findings

1. This repository should use heuristic relationship extraction and cluster formation as the backbone of the system, and reserve LLM synthesis for bounded candidate sets, because the corpus already exposes explicit signals such as tags, cites, related, and changed files that are cheap to compute and safer than whole-corpus prompting.
2. A document-summary index pattern is the best first retrieval layer for active synthesis in this repo, because it stores one summary per item, retrieves whole documents by summary relevance, and fits the file-based constraints better than introducing vector infrastructure before W-0025 is revived.
3. A D3 force-directed graph over graph JSON is the strongest default knowledge-map renderer for this repository's current corpus and publishing model, while Mermaid is better treated as a low-cost derivative for small subgraphs rather than as the primary full-corpus map.
4. Graph JSON should be the primary runtime artifact and JSON-LD should be a secondary export, because graph JSON is the simplest structure for Python transforms and browser rendering, while JSON-LD adds semantic interoperability without being the easiest working format for static graph navigation.
5. The best-aligned initial trigger design is push for deterministic artifact refresh, schedule for weekly or batched distillation, and workflow_dispatch for ad hoc synthesis, because the repository already uses those patterns successfully and each trigger aligns with a distinct cost and freshness profile.
6. STORM-like multi-perspective synthesis should be a selective second-stage method for broad, ambiguous, or contradiction-heavy clusters rather than the default path, because it improves organization and breadth but carries materially higher reasoning and orchestration cost than deterministic selection plus ordinary synthesis prompts.
7. The repository should publish new synthesized insights as ordinary completed synthesis items plus targeted learnings.md updates, not as a new content store, because the current item schema already supports citations, confidence, provenance, and version history for exactly this kind of output.
8. A Letta-style memory split is the right mental model for the synthesis interface, with summaries and recent digests acting as working memory and the full corpus acting as searchable archival memory, because that preserves provenance and keeps prompt size bounded without requiring a standing service.

Assumptions

• [assumption] The corpus will remain modest enough through the next implementation phase that graph JSON generation, per-item summary refresh, and static D3 layout computation all fit comfortably inside a normal GitHub Actions run. Justification: the existing site-build workflow already regenerates derived artifacts on push, and D3 explicitly supports offline computation of static layouts. [source: GitHub workflow: Build and Deploy Research Site d3js.org
• [assumption] Existing cites and related metadata can be populated consistently enough to become useful edge types in the first graph build without adding a new annotation workflow first. Justification: recent completed items already use these fields materially, so the graph can start from them and improve over time. [source: Mitchell (2026) Is knowledge scaffolding an established concept within context engineering for Large Language Models and AI agents? github.com

Analysis

The decisive design move is to separate structure generation from interpretation rather than trying to make one artifact serve both perfectly.

That separation produces a coherent stack: Python scripts extract summaries and edges into JSON, D3 renders graph navigation from that JSON, and a later synthesis workflow consumes those same artifacts to decide what deserves LLM reasoning.

The most important trade-off is freshness versus cost, not simple automation versus manual work.

Refreshing deterministic artifacts on every push gives immediate navigability at low risk, while delaying expensive distillation to a weekly schedule keeps the active requirement without paying STORM-like costs after every single completion.

The graph-format choice follows the same logic: graph JSON is operationally simplest, D3 is interaction-rich enough for the full map, Mermaid remains useful for small embedded cluster views, and JSON-LD preserves future interoperability without complicating the primary pipeline.

This also keeps the system legible to reviewers, because every synthesized conclusion can point back to stable source items, summary artifacts, and explicit cluster manifests instead of disappearing into an opaque vector store or a one-shot whole-corpus prompt.

Risks, Gaps, and Uncertainties

• The recommendation is strongest on architecture shape and weakest on exact cluster-quality thresholds, because the sources support layered selection and synthesis more clearly than they specify the precise heuristic cutoffs that will work best for this corpus.
• The evidence for Open Knowledge Maps and Obsidian Publish supports the value of graph navigation, but it does not by itself prove which specific interaction design will be optimal for this repository's tag density and link structure.
• STORM provides strong evidence that multi-perspective pre-writing improves broad synthesis, but it is an article-generation system rather than a repository-governance workflow, so the recommendation to use it selectively remains an architectural inference rather than a direct product-level prescription.
• Embeddings may still become necessary later if heuristic cluster formation degrades as the corpus grows, but current evidence does not justify making them part of the baseline architecture before W-0025 is revived.

Open Questions

• At what corpus size or cluster-count does heuristic cluster formation stop being good enough and require persisted embeddings or another semantic-retrieval layer?
• Which edge types should be weighted most heavily in the first graph: cites, related, shared tags, or shared extracted concepts from summaries?
• Should active weekly digests update learnings.md automatically, open a draft synthesis item automatically, or do both depending on whether the new insight is thematic or transient?

sources

• [x] Shao et al. (2024) Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models
• [x] LlamaIndex Document Summary Index
• [x] GitHub Docs: Events that trigger workflows
• [x] GitHub workflow: Build and Deploy Research Site
• [x] GitHub workflow: Research Loop
• [x] Letta Docs: Archival memory
• [x] Obsidian Publish
• [x] Mermaid Flowcharts - Basic Syntax
• [x] D3 Force Simulation
• [x] JSON-LD
• [x] Open Knowledge Maps
• [x] Mitchell (2026) Cross-item synthesis: methodology and tooling for extracting meta-insights from the research corpus
• [x] Mitchell (2026) Exploration-synthesis gap: why people in explore mode fail to synthesise others' work, and whether agent synthesis can close the gap
• [x] Mitchell (2026) Application Programming Interface context hubs, Retrieval-Augmented Generation, and the Model Context Protocol
• [x] Mitchell (2026) Is knowledge scaffolding an established concept within context engineering for Large Language Models and AI agents?
• [x] Mitchell (2026) What principles and governance practices enable sustainable, high-quality software development with Artificial Intelligence coding agents?