How can findings from OpenFactCheck, Loki, FActScore, gpt-oss-safeguard, and…

Research Question

How can the findings from research into OpenFactCheck, Loki, FActScore, gpt-oss-safeguard, and Barnum statement identification techniques be synthesised into concrete, actionable improvements to the automated review process (research-review-prompt.md) in this repository, specifically targeting factual precision, policy compliance, and output quality?

Findings

Executive Summary

A pragmatic sequence is therefore: rewrite research-review-prompt.md first, insert a deterministic lint step into research-review.yml second, strengthen research-prompt.md with a positive specificity contract third, and treat OpenFactCheck, FActScore, Loki, and safeguard-style policy classifiers as sampled or optional future backlog items rather than the immediate default gate.

research-review-prompt.md already checks citation binding, speculation control, prose quality, and logical coherence, but it does not make atomic-claim coverage, concrete-anchor absence, or source recency first-class review failures.

The five prerequisite items converge on a layered architecture: prompt-side specificity contracts to prevent hollow prose, deterministic linting to catch repeatable defects cheaply, and sampled claim- or policy-level model assistance only where a bounded escalation path is justified.

The highest-value sequence is therefore: rewrite research-review-prompt.md first, insert a deterministic lint step into research-review.yml second, strengthen research-prompt.md with a positive specificity contract third, and treat OpenFactCheck, FActScore, Loki, and safeguard-style policy classifiers as sampled or optional future backlog items rather than the immediate default gate.

Key Findings

1. research-review-prompt.md currently enforces citation, speculation, prose-quality, and coherence checks, but it does not explicitly fail drafts for bundled multi-proposition claims, low-specificity generic-sounding sentences, or stale evidence on time-sensitive topics.
2. The strongest prompt-only upgrade is to extend research-review-prompt.md Step 1 with an atomic-claim completeness check, Step 3 with an explicit check for Barnum statements, low-specificity sentences that sound analytical without adding decision-useful detail, and Step 4 with a recency check for live or fast-changing claims.
3. The highest-impact low-infrastructure tooling change is a deterministic pre-review linter that checks exact source-link parity, checked-source coverage, acronym expansion, filler-phrase bans, concrete-anchor heuristics, and simple date-based recency rules before the semantic review pass runs.
4. research-prompt.md should prevent defects upstream by requiring each analytical sentence to name a concrete actor, mechanism, metric, disagreement, or decision consequence and by adding a short support-critical claim inventory before final Findings are drafted.
5. OpenFactCheck and FActScore both support a future support-critical claim-audit path, but the repository's own prior syntheses judge each tool family better as an offline diagnostic or sampled escalation path than as a mandatory per-item inline gate.
6. Loki adds useful human-facing evidence discovery and gpt-oss-safeguard adds policy-conditioned reasoning, but both are better treated as later-stage or sampled integrations because Loki depends on live evidence retrieval and safeguard models do not fit the default GitHub-hosted review runner.
7. The current evidence supports a pragmatic priority order of review-prompt rewrite first, deterministic lint second, research-prompt rewrite third, sampled support-critical claim verification fourth, and policy-classification or live-web evidence escalation after those foundations exist.

Assumptions

• Assumption: The repository will continue producing mixed-domain synthesis items rather than switching to one bounded verification corpus. Justification: the current prompts and completed corpus are oriented around multi-source synthesis rather than one canonical knowledge base.
• Assumption: Barnum-candidate heuristics should begin as flags or warnings rather than hard failures until a repository-specific labeled sample exists. Justification: the Barnum synthesis found no direct benchmark for research-prose prevalence, so low-cost detection is justified before blocking enforcement is calibrated.
• Assumption: Time-sensitive claims can be identified with a simple author-supplied note or date heuristic before a more advanced retrieval layer exists. Justification: the current prompt already requires source enumeration, so adding a lightweight recency signal is operationally plausible without new infrastructure.

Analysis

The main trade-off is between coverage breadth and workflow friction, and the reviewed evidence favors staged controls rather than a single universal verifier.

For research-review-prompt.md, the exact low-cost change is to add one citation-discipline bullet for atomic-claim completeness, one remove-ai-slop bullet for Barnum statements and concrete-anchor absence, and one peer-reviewer bullet for stale or weakly dated evidence on time-sensitive claims.

Inside .github/workflows/research-review.yml, the exact tooling change is to run a deterministic lint script before the Copilot invocation so that missing checked sources, source-link mismatches, acronym misses, filler phrases, simple Barnum candidates, and date or recency omissions fail fast without consuming a full semantic review pass.

Upstream in research-prompt.md, the exact change is to add a positive specificity contract near the Findings instructions and a short support-critical claim inventory near the end of investigation, because that combination prevents both low-specificity prose and over-bundled facts before review begins.

The strongest rival remedy is to move directly to OpenFactCheck, FActScore, Loki, or safeguard-model integration, but that route is weaker as a first step because every reviewed tool family assumes a narrower corpus, heavier infrastructure, or a later-stage escalation workflow than the repository currently operates.

That is why the priority matrix is asymmetric: review-prompt rewrite is low effort and high uplift, deterministic lint is low-to-medium effort and high uplift, research-prompt prevention is medium effort and high uplift, sampled claim verification is medium-to-high effort and medium uplift, and full policy or live-web semantic escalation is high effort with more conditional uplift.

Risks, Gaps, and Uncertainties

• Direct benchmark evidence for Barnum detection in research prose is still missing, so any first deterministic concrete-anchor rule should begin as a warning or low-severity failure until the repository calibrates it on labeled examples.
• A simple source-recency rule can catch obviously stale citations, but it cannot by itself prove factual invalidity, so recency should qualify confidence and trigger review rather than serve as a stand-alone truth test.
• Sampled support-critical claim auditing can improve factual precision, but it will still miss omission-heavy or non-sampled errors unless the repository later adds a bounded selection policy and calibration data.
• Policy-model integration remains infrastructure-bound until the repository has an external inference surface or a clearly scoped sampled workflow, so policy compliance is best kept rubric-first in the short term.

Open Questions

• Draft backlog item: Build deterministic research review linter - add a Python pre-review parser and checker that fails fast on source-link mismatches, unchecked seeded sources, acronym misses, filler phrases, simple Barnum candidates, and date or recency omissions before the Copilot review step.
• Draft backlog item: Add support-critical claim audit harness - decompose Executive Summary and Key Findings into atomic claims and run sampled offline support checks inspired by FActScore and OpenFactCheck rather than a full-document inline gate.
• Draft backlog item: Calibrate Barnum detection on the completed corpus - label a repository sample for concrete-anchor absence and compare deterministic heuristics with Large Language Model as judge scoring before making Barnum checks blocking.

sources

• [x] Research repository research-review prompt - primary target document for review-rubric changes
• [x] Research repository research prompt - upstream generation prompt for prompt-only prevention changes
• [x] Research repository research-review workflow - current automated review execution path and insertion point for deterministic tooling
• [x] Wang et al. (2024) OpenFactCheck: A Unified Framework for Factuality Evaluation of Large Language Models (LLMs) - primary OpenFactCheck paper
• [x] Min et al. (2023) FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation - primary FActScore paper
• [x] Inan et al. (2023) Llama Guard: Large Language Model (LLM)-based Input-Output Safeguard for Human-AI Conversations - primary open-weight safeguard comparator
• [x] Sharma et al. (2023) Towards Understanding Sycophancy in Language Models - primary sycophancy evidence used to qualify Barnum detection
• [x] Augenstein et al. (2024) Factuality Challenges in the Era of Large Language Models and Opportunities for Fact-Checking - cross-cutting survey on LLM factuality and fact-checking limits
• [x] Research repository OpenFactCheck AI fact-checking pipeline - prerequisite synthesis on OpenFactCheck workflow fit
• [x] Research repository Loki fact-checking journalists moderation - prerequisite synthesis on Loki workflow fit
• [x] Research repository FActScore precision scoring atomic claims - prerequisite synthesis on FActScore scope and limits
• [x] Research repository gpt-oss-safeguard policy enforcement open weight - prerequisite synthesis on policy-classification fit
• [x] Research repository Barnum statements AI responses theory practice - prerequisite synthesis on low-specificity output failures
• [x] Research repository automated claim verification academic literature - adjacent synthesis on support-critical claim verification
• [x] Research repository human review volume bottleneck rubber stamp - adjacent synthesis on review-load limits
• [x] Research repository research loop quality prompt engineering - adjacent synthesis on prompt-level quality controls
• [x] Research repository Large Language Model (LLM) as judge pipeline validation checkpoints - adjacent synthesis on layered evaluation and judge limits