Research agenda curation

Research Question

How should the research backlog be maintained and prioritised to ensure balanced coverage of important domains, detect over-concentration in one area (research drift), and surface high-value gaps — rather than accumulating items that are interesting but strategically unfocused?

Findings

Executive Summary

The research backlog for davidamitchell/Research currently lacks a domain map, a principled prioritisation rubric, and a drift detection mechanism — three interlocking deficiencies that cause the autonomous research loop to process items in creation order rather than strategic priority order. Priority inflation (88% of 25 items rated medium) and domain concentration (44% of backlog in Research Tooling & Delivery) are the two observable symptoms. The remedy is a five-domain taxonomy, a four-dimension prioritisation rubric, a 40%-concentration drift detection threshold, and a monthly agenda review triggered by workflow_dispatch and reported as a GitHub issue. All tooling can be implemented by extending src/research/cli.py with a research agenda command using only existing dependencies and parsed YAML fields.

Key Findings

1. Priority inflation renders the priority field ineffective. 88% of 25 backlog items are rated medium. The autonomous loop's selection algorithm degrades to oldest-first when all items are equally prioritised. [high confidence — direct measurement]

2. D4 Research Tooling & Delivery has drifted to 44% of backlog. 11 of 25 items cluster around tooling, interface, delivery, and transcript fetching — crowding out AI Strategy (D1) and Agentic Systems (D2) items more directly tied to professional decision-making. [high confidence — direct count]

3. Five domains cover the corpus without gaps. D1 AI Strategy & Governance; D2 Agentic Systems & Architecture; D3 Cognitive Science & Foundations; D4 Research Tooling & Delivery; D5 Knowledge Management & Process. All 48 items map to exactly one primary domain. [medium confidence — taxonomy is an inference; some items straddle domains]

4. The prioritisation rubric uses four dimensions: decision dependency, dependency unblocking, gap severity, input availability. Scored 1–3 each; total 10–12 = high, 6–9 = medium, 4–5 = low. This produces a distribution where high requires ≥3 in at least two dimensions — genuinely rare by design. [medium confidence — synthesised from JTBD, OKR, PARA; not empirically validated]

5. D1 (AI Strategy & Governance) should maintain a protected minimum of ≥2 backlog items. As the domain most directly tied to the owner's professional context, it should trigger a coverage alert if it drains to zero during loop processing. [medium confidence — professional context inference]

6. D2 (Agentic Systems) is currently under-represented. 2 of 25 backlog items despite 4 completed items and active professional relevance. No new agentic items have been queued — this is a gap requiring agenda intervention. [high confidence — direct count]

7. Drift detection threshold: ≥40% domain concentration in items added in the last 30 days. Secondary signal: last 5 completed items all from the same domain. The 40% threshold tolerates intentional focused sprints while catching unintentional drift. [medium confidence — calibrated estimate, not empirically validated]

8. Monthly review cadence matches corpus turnover rate. At 3 items/day on weekdays, the 25-item backlog turns over in ~8 weeks. Monthly reviews catch imbalances before the next cycle. [high confidence — arithmetic]

9. The research agenda CLI command is implementable without new dependencies. All required fields (tags, priority, added, completed, blocks) are already parsed by ResearchItem.from_file(). The command requires only a domain-map constant and reporting logic added to src/research/cli.py. [high confidence — direct source inspection]

10. The rubric must be embedded at item-addition time, not only at review time. Priority inflation occurs because the default is medium and there is no friction at creation. The CLI template in src/research/cli.py and the AGENTS.md item-addition instructions should reference the rubric. [medium confidence — behavioural inference]

11. workflow_dispatch + GitHub issue is the correct delivery mechanism for agenda reports. This is the owner's access model and is explicitly prescribed in AGENTS.md. No new credentials required. [high confidence — AGENTS.md direct]

12. Open Questions in completed items are an under-used backlog input. Multiple completed items contain follow-on questions not yet added to backlog. The monthly review checklist should scan Open Questions sections systematically. [medium confidence — qualitative observation]

Assumptions

• Assumption: The five-domain taxonomy is stable for this repo. Justification: The corpus has 48 items with clear tag clusters; the five domains emerged from bottom-up tag analysis and align with the owner's described professional context (NZ financial services, SWE, personal intellectual development). A sixth domain (e.g., "Emerging Technology Trends") is possible but no tag cluster currently supports it.
• Assumption: 3 items/day processing rate is sustained on weekdays. Justification: This is the configured scheduled rate in research-loop.yml. Actual rate may vary based on item complexity and loop failures.
• Assumption: The 40% drift threshold is appropriate for this corpus. Justification: Calibrated from first principles based on corpus size and processing rate; not empirically tested. Should be reviewed after 3 months of operation.

Analysis

The core finding is that the backlog's two observable failures (priority inflation and domain concentration) share a single root cause: there is no friction at item-creation time. When adding an item is easy and there is no prompt to apply a rubric or check domain balance, items accumulate with default priority and in the domain of the current session's focus. The remedy must be applied upstream (at creation) and downstream (at monthly review), not only when the loop selects the next item.

JTBD, OKR, and PARA each contribute a different constraint to the prioritisation rubric:

• JTBD contributes the "decision dependency" dimension — priority derives from downstream use, not topic interest
• OKR contributes the concept of top-down quarterly constraints — a research OKR can override the bottom-up rubric for specific items
• PARA contributes the insight that "Project-level" research (needed for current active work) should always be high, regardless of other dimensions

The four-dimension rubric synthesises all three: it requires both downstream use (JTBD/PARA) and availability (pragmatic) and connectivity (OKR-like unblocking) to reach high. Items that are merely interesting score at most 7–8 (medium).

The trade-off between drift detection sensitivity and false-positive rate is managed by the 40% threshold and the 30-day window. A focused research sprint (e.g., "complete all transcript-fetching approaches") should not be penalised — it is intentional concentration. The distinction is whether the concentration was deliberate (items were added in a burst to complete a known cluster) or inadvertent (items drifted in one direction over time without recognition). The monthly review checklist includes a question to distinguish these cases.

Risks, Gaps, and Uncertainties

• The 40% drift threshold is calibrated from first principles rather than historical data. It may need adjustment after 3 months of operation. The threshold should be a configurable parameter, not a hard-coded constant.
• The five-domain taxonomy does not cover all possible future additions. As the repo owner's interests evolve, new domains may emerge. The domain map constant in src/research/cli.py should be easy to update without code changes — consider moving it to config/sources.yaml or a new config/domains.yaml.
• Priority inflation is a recurrence risk. Even with the rubric embedded in the template, agents adding items will sometimes skip the rubric under time pressure. A CI check that flags items with no explicit rubric justification (e.g., missing a priority_rationale field) would be stronger than a template comment, but adds schema complexity.
• The five-domain taxonomy based on primary tags may incorrectly assign some items. agent-lsp-policy-enforcement could belong to D1 (governance), D2 (agentic systems), or D4 (tooling) depending on how it is read. The domain assignment logic in the agenda command should use tag intersection (item tags ∩ domain primary tags → highest overlap domain) rather than single-tag lookup.
• Open Questions in completed items have never been systematically harvested. The monthly review checklist requires a human (or agent) to read Open Questions sections and compare against the backlog. This is manual and error-prone; a future CLI command could automate it.

Open Questions

• Should the domain map be moved to a configuration file (config/domains.yaml) rather than a constant in src/research/cli.py? Would allow the owner to adjust domain boundaries via the GitHub web editor without code changes. → New backlog item: moderate priority.
• Should there be a priority_rationale field in the research item front-matter to document why an item was assigned a given priority? Would provide an audit trail and enforce rubric application. → AGENTS.md update consideration.
• Should a CI check fire when a PR adds a new backlog item without applying the prioritisation rubric? This would be a structural check, not a content check. → Could be implemented as a GitHub Actions step in ci.yml.
• As the backlog shrinks toward 0 (loop consuming items faster than they are added), should the agenda tool produce a "backlog replenishment" alert? At <5 items, the owner should be prompted to run a backlog-filling session. → New backlog item or AGENTS.md update.
• Is there a quarterly "research OKR" worth introducing formally — e.g., a Research/objectives/ directory with quarterly .md files? This would provide top-down priority constraints that override the bottom-up rubric for specific items. → Worth exploring once the basic rubric and review mechanism are in place.

sources

• [x] AGENTS.md — When the Backlog Is Empty section; research item lifecycle; distinction between repo backlog and research backlog
• [x] Research/backlog/2026-02-27-sources-of-research.md — orthogonal question: where new items come from; this item answers what to do with them once they arrive
• [x] Research/completed/2026-02-27-research-backlog-vs-repo-improvement-backlog.md — the documented distinction between the two backlogs; apply the same clarity to agenda management
• [x] BACKLOG.md — existing prioritisation discipline for repo improvements; patterns to adapt
• [x] Personal Knowledge Management (PKM) literature on research agenda management: Tiago Forte's PARA method, progressive summarisation as a triage tool; relevant to deciding what to research vs. what to archive
• [x] Jobs-to-be-done framework (Christensen): structured approach to defining value and priority by outcome rather than by topic interest — applicable to research prioritisation
• [x] OKR (Objectives and Key Results) methodology: assess whether a quarterly research OKR (e.g., "understand the NZ AI regulatory landscape well enough to advise on compliance") provides useful top-down prioritisation constraints
• [x] Research/backlog/2026-03-02-research-quality-assurance-methodology.md — identifies "applicability testing" as a quality dimension; agenda curation is the upstream version: only add items where the knowledge would be applicable
• [x] src/research/item.py and src/research/cli.py — existing YAML front-matter schema and CLI; extension points for agenda command