What automated claim verification approaches against scientific literature…

Research Question

What automated claim verification approaches against scientific literature, specifically arXiv preprints, are used in research synthesis systems, what search strategies maximise recall and precision for claim-to-paper matching given a natural-language claim, and what is the minimum-viable verification workflow that an Artificial Intelligence (AI) research agent using the arxiv_mcp_server Model Context Protocol (MCP) tool can execute to verify that a support-critical claim, one a Key Finding directly depends on, is supported by a specific primary paper, and what should happen when verification fails (downgrading claim label from [fact] to [inference] with explanation)?

Findings

Executive Summary

A minimum-viable arXiv verification workflow should separate paper retrieval from support judgment and should leave a support-critical claim marked as [fact] only after one identified paper explicitly supports the same proposition. Scientific claim-verification systems consistently decompose the task into retrieval, evidence selection, and support or refute judgment, which implies that paper discovery and evidence interpretation are distinct failure surfaces in a research-agent workflow. Because open-domain studies find that lexical retrieval tends to maximize precision while semantic retrieval improves recall, the current arxiv_mcp_server is best used as a lexical-first verifier surface with bounded candidate inspection rather than as a complete literature-search solution. Large Language Models remain too hallucination-prone and too brittle to invent papers or act as sole verifiers, so the safest role for the model is structured support or refute assessment over already retrieved candidate papers, with downgrade to [inference] whenever the paper match or support threshold fails.

Key Findings

1. Scientific claim-verification systems from FEVER through SciFact, VerT5erini, and MultiVerS consistently decompose verification into retrieval, evidence selection, and support or refute judgment, even when later models integrate some stages more tightly.
2. Open-domain verification results show that retrieval quality materially changes final verification quality, with lexical methods favoring precision, semantic methods favoring recall, and hybrid retrieval giving the strongest overall pattern when the system can support it.
3. arxiv_mcp_server exposes global paper search, per-paper download, and paper reading, while its semantic search only works over locally downloaded papers and therefore cannot replace the initial global search step.
4. Large Language Models are best treated as unsuitable primary literature finders or single-shot verdict generators, because fabricated references, perturbation brittleness, and multiple-choice position bias all remain well-documented failure modes in academic and fact-verification settings.
5. A support-critical claim should remain [fact] only when one identified arXiv paper states the same or materially equivalent proposition, because topic-level similarity or partial support is not strong enough to preserve verified status in later synthesis.
6. A bounded workflow that verifies only a small number of support-critical claims per item and inspects only a small top-ranked candidate set is the right minimum-viable compromise for this repository.
7. When no plausible paper match appears, when the best candidate is only topically related, or when support is partial or contradictory, the workflow should downgrade the claim to [inference] and record the query, candidate, and failure reason instead of preserving false certainty.

Assumptions

• A top-ten lexical arXiv search is usually wide enough to surface a plausible candidate paper for a well-phrased support-critical claim.
• Abstracts are sufficient for rejection and triage decisions, while positive retention of [fact] may require full-text reading when qualifiers matter.
• The repository can tolerate the search and read cost for at most five support-critical claims per item without distorting session time budgets.

Analysis

A hybrid retrieval stack would be stronger than a lexical-only stack, but the available tool surface does not provide global semantic retrieval, so the practical recommendation optimizes precision and inspectability instead of claiming best-possible recall. The verification literature and the repository's prior synthesis work point in the same direction: retrieval alone is not enough, because later synthesis can still collapse near-miss papers into one overconfident claim unless support is checked at the paper and proposition level. The downgrade rule is intentionally conservative because a missed fact can survive as a labeled inference, while a mislabeled fact contaminates later reasoning as if it were verified ground truth.

Risks, Gaps, and Uncertainties

• The empirical retrieval evidence is strongest in biomedical or health-adjacent corpora, so generalization from PubMed-oriented benchmarks to every arXiv domain is plausible but not directly demonstrated here.
• No primary study directly measured author or title lookup against claim-keyword search on arXiv, so any claim that title or author lookup is best should be treated as a workflow heuristic rather than as benchmarked fact.
• Abstract-only verification can miss qualifiers or boundary conditions that appear later in the paper, which means some retained [fact] judgments will remain weaker than full-text-confirmed support.

Open Questions

• Would downloading a larger candidate pool and then running local semantic search improve recall enough to justify the added time and complexity?
• Should a later version of the workflow use citation-graph expansion when the top lexical hits are close but none makes the claim explicitly?
• What lightweight audit step best distinguishes a genuine verification failure from a poor query formulation before a support-critical claim is downgraded?

sources

• [x] Thorne et al. (2018) FEVER: a Large-scale Dataset for Fact Extraction and VERification - foundational fact-verification benchmark and retrieval-plus-entailment pipeline design
• [x] Wadden et al. (2020) Fact or Fiction: Verifying Scientific Claims - SciFact task definition, abstract retrieval, rationale selection, and support/refute labels for scientific claims
• [x] Pradeep et al. (2021) Scientific Claim Verification with VerT5erini - pretrained sequence-to-sequence retrieval, sentence selection, and label prediction on SciFact
• [x] Wadden et al. (2022) MultiVerS: Improving scientific claim verification with weak supervision and full-document context - full-document context and multitask rationale-plus-label prediction
• [x] Vladika and Matthes (2024) Comparing Knowledge Sources for Open-Domain Scientific Claim Verification - retrieval precision and recall trade-offs across Best Matching 25 (BM25) and semantic search
• [x] Zheng et al. (2024) Evidence Retrieval is almost All You Need for Fact Verification - hybrid retrieval evidence and the centrality of evidence selection to final verification quality
• [x] Dmonte et al. (2024) Claim Verification in the Age of Large Language Models - overview of Large Language Model-based verification pipelines and retrieval-augmented patterns
• [x] Chelli et al. (2024) Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis - empirical evidence on hallucinated citations in academic literature search tasks
• [x] Agrawal et al. (2024) Do Language Models Know When They're Hallucinating References? - consistency-check evidence for detecting fabricated references after generation
• [x] Zheng et al. (2024) Large Language Models Are Not Robust Multiple Choice Selectors - position-bias evidence against multiple-choice-only verifier prompts
• [x] Huwiler et al. (2025) FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models - robustness evidence for fact-verification brittleness under small perturbations
• [x] de Vries et al. (2024) The perils and promises of fact-checking with large language models - retrieval-backed Large Language Model agent accuracy remains inconsistent
• [x] blazickjp (2026) arxiv-mcp-server GitHub repository - tool documentation for search, download, local semantic search, and rate limits