Trusting Trust and AI Corpus Contamination

Research Question

Ken Thompson's "Trusting Trust" argument shows that you cannot verify a compiler by reading its source code if the compiler was compiled by a compromised toolchain — the contamination lives in the binary, not the source. What is the web-scale analogue for Artificial Intelligence (AI)-generated content, and what does it mean for epistemology, knowledge verification, and trust when roughly half of all text on the web is now AI-generated and that proportion is growing?

Findings

Executive Summary

[inference] The public web now behaves less like a neutral evidence commons and more like a partially recursive corpus whose provenance cannot be validated by reading any single page in isolation. (Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025) ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)

[inference] Thompson's trusting-trust argument is therefore best applied here as a warning about recursive corpus contamination rather than as a claim that the web contains a literal compiler-style backdoor. Later models and later readers increasingly consume outputs partly produced by earlier models, which weakens local source inspection as a test of independence. (Sources: Ken Thompson, "Reflections on Trusting Trust" (1984) ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024) ; Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (2024)

[fact] The best accessible prevalence evidence does not establish that half of all web text is AI-generated, but it does support a large current share, with active-web estimates around 30-40% and some sampled article sets already near parity. (Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025)

[inference] The practical result is epistemic as well as technical: citation and agreement remain useful, but they now carry more weight when they demonstrate provenance and source independence rather than repeated fluency across webpages. (Sources: Internet Encyclopedia of Philosophy, "Epistemic Circularity" ; Ken Thompson, "Reflections on Trusting Trust" (1984)

Key Findings

1. [inference] Thompson's trusting-trust argument maps most closely to recursive corpus contamination because both involve upstream corruption that reproduces across generations while remaining largely invisible when a reviewer inspects only the final visible artifact. Confidence: high. (Sources: Ken Thompson, "Reflections on Trusting Trust" (1984) ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)
2. [fact] The strongest publicly accessible prevalence evidence shows that AI-generated text is already a large share of the public web, but the underlying studies support a bounded range rather than a settled claim that half of all web text is synthetic. Confidence: medium. (Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025)
3. [fact] Shumailov and colleagues show that recursive training on generated data causes models to lose tail information and drift away from the original data distribution, which means corpus contamination can alter what future systems are capable of representing, not just what they happen to retrieve. Confidence: high. (Source: Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)
4. [fact] Gerstgrasser and colleagues show that recursive contamination is not mechanically inevitable under every data regime, because retaining original real data alongside synthetic data materially changes the outcome and can bound collapse in their experiments. Confidence: high. (Source: Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (2024)
5. [inference] The epistemic danger is that apparently independent webpages can become practically circular evidence when they are all descended from the same generative loop, so citation count and fluent agreement stop being reliable proxies for independent confirmation. Confidence: high. (Sources: Internet Encyclopedia of Philosophy, "Epistemic Circularity" ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)
6. [inference] Prompt injection and corpus contamination are structurally related because both exploit the model's inability to distinguish trusted instructions from untrusted language-shaped inputs, but prompt injection acts at inference time while corpus contamination degrades the evidence and training base over longer horizons. Confidence: high. (Sources: Open Worldwide Application Security Project (OWASP) Large Language Model (LLM)01:2025 Prompt Injection ; Abdelnabi et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (2023) ; OWASP LLM04:2025 Data and Model Poisoning
7. [inference] Document-level AI detection cannot solve the trusting-trust-style problem because detecting one synthetic page does not reveal whether the broader corpus, retrieval chain, or training lineage behind a claim is independent and trustworthy. Confidence: high. (Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025) ; Internet Encyclopedia of Philosophy, "Epistemic Circularity"
8. [inference] The most defensible practical response is to treat open-web knowledge more like a software supply chain by preferring primary sources, preserving trusted human-generated data reservoirs, recording provenance, and requiring stronger human review for high-stakes grounded outputs. Confidence: high. (Sources: Ken Thompson, "Reflections on Trusting Trust" (1984) ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024) ; Open Worldwide Application Security Project (OWASP) Large Language Model (LLM)01:2025 Prompt Injection

Assumptions

• [assumption] Assumption: Provenance-verified human-generated data remain a cleaner reference class than heavily synthetic corpora for preventing recursive degradation. Justification: The Nature paper explicitly argues that access to real human-produced data becomes increasingly valuable as generated content pollutes the internet. Source: Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)
• [assumption] Assumption: The article-sample and active-page prevalence studies are sufficiently independent to justify a directional conclusion about substantial contamination. Justification: They use different methods and still converge on a large synthetic share rather than a trivial one. Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025)

Analysis

[inference] The structural part of the case is stronger than the quantitative part because the Thompson lecture and the recursive-training papers independently support the claim that integrity can fail upstream while downstream outputs remain superficially coherent. (Sources: Ken Thompson, "Reflections on Trusting Trust" (1984) ; Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024) ; Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (2024)

[inference] The prevalence evidence is less certain because there is no authoritative whole-web census, but the accessible studies and NewsGuard's tracking still show that contamination pressure is already large enough to matter operationally for research and grounding workflows. (Sources: Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025) ; Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025) ; NewsGuard AI Tracking Center

[inference] The security material helps separate layers of the problem: prompt injection is the short-horizon exploit form of a broader trust-boundary failure, while corpus contamination is the long-horizon evidence-base form. That distinction is why provenance review and primary-source preference are more useful controls here than document-level authorship checks alone. (Sources: Open Worldwide Application Security Project (OWASP) Large Language Model (LLM)01:2025 Prompt Injection ; Abdelnabi et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (2023) ; OWASP LLM04:2025 Data and Model Poisoning

Risks, Gaps, and Uncertainties

• No public source in this review provides a definitive whole-web percentage for AI-generated text.
• The accessible prevalence studies depend on imperfect detection or marker methods.
• This item does not establish how much contamination current frontier-model pipelines actually tolerate before performance degrades materially.
• Mixed human-AI authorship complicates binary categories such as "human" versus "AI-generated."
• The philosophical mapping from epistemic circularity to citation practice is strong but not mathematically formalized here.

Open Questions

• What provenance standards would let Retrieval-Augmented Generation (RAG) systems rely on open-web material without inheriting circular evidence loops?
• Can search and retrieval systems rank source independence and provenance quality, not just relevance and authority signals?
• How quickly are synthetic-content farms reshaping citation graphs, search results, and future training corpora in practice?
• What minimum reservoir of verified human-generated data is needed to keep recursive training from erasing tail knowledge in large production systems?

sources

• [x] Ken Thompson, "Reflections on Trusting Trust" (1984)
• [x] Shumailov et al., "AI models collapse when trained on recursively generated data" (Nature, 2024)
• [x] Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (2024)
• [x] Spennemann, "Delving into the quantification of AI-generated content on the internet" (2025)
• [x] Graphite Common Crawl analysis, "More Articles Are Now Created by AI Than Humans" (2025)
• [x] NewsGuard AI Tracking Center
• [x] Open Worldwide Application Security Project (OWASP) Large Language Model (LLM)01:2025 Prompt Injection
• [x] OWASP LLM04:2025 Data and Model Poisoning
• [x] Abdelnabi et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (2023)
• [x] Simon Willison, "Model Context Protocol (MCP) prompt injection" (2025)
• [x] Internet Encyclopedia of Philosophy, "Epistemic Circularity"
• [x] Related completed item: Research/completed/2026-03-15-prompt-injection-threat-landscape.md