When Retrieval-Augmented Generation source documents change after agent build…

Research Question

When the source documents indexed in a Retrieval-Augmented Generation (RAG) pipeline change after an agent has been built and tested, what failure modes and behavioral regressions can result in production, and what practices, covering document versioning, behavioral baseline testing, Configuration Management Database (CMDB)-style dependency registration of agent-to-document relationships, and Information Technology Infrastructure Library (ITIL)-inspired change-management governance, exist to detect, govern, and mitigate these regressions?

Findings

Executive Summary

Post-deployment document changes in Retrieval-Augmented Generation systems act like dependency updates: they can change the retrieved evidence, citations, and downstream agent behavior even when model weights and prompts stay fixed.

The highest-confidence failure mechanisms are stale or orphaned indexed content, duplicate or conflicting articles, chunk or structure changes that alter ranking, and multi-query retrieval plans that change which evidence reaches the model.

Current evaluation frameworks can catch many of these regressions through golden datasets, retrieval metrics, and traced production monitoring, but they do not turn the corpus version into a first-class governed dependency on their own.

The best-supported operational response in this evidence base is to treat the corpus and index as deployable artifacts with version identifiers, staged promotion controls, rollback paths, and registry links from each agent to the corpus version it was tested against.

Key Findings

1. Retrieval-Augmented Generation and agentic retrieval systems can change behavior after document updates because retrieved passages and query plans are part of the model input at inference time, not fixed compile-time assets.
2. Retrieval regressions arise from more than factual edits, because failed deletion handling, renamed paths, duplicate articles, non-self-contained articles, and chunking changes can all alter what evidence is retrieved or summarized.
3. The most operationally important behavioral regressions are stale answers, missing facts, blended or contradictory answers, citation drift, and changed workflow choices when altered grounding changes what the agent considers relevant.
4. LangSmith, Ragas, and TruLens collectively provide datasets, experiment comparison, retrieval metrics, ground-truth checks, and traced production monitoring, which together form the practical ingredients for corpus-change regression testing.
5. None of the inspected evaluation frameworks or platform documents automatically turns the document corpus into a first-class dependency record, so teams need explicit corpus version identifiers, trace metadata, or registry entries to know what was tested and what is live.
6. Search-infrastructure controls such as aliases, blue-green deployment, replicas, and recoverable indexers make staged rollout and recovery technically feasible for corpus-bearing indexes, so teams can adapt those primitives into corpus-version promotion and rollback workflows even though the cited sources do not document a turnkey RAG pattern.
7. Repository dependency-mapping work and ServiceNow knowledge-governance guidance together suggest a governance template for linking agents to governed corpus assets, but the accessible sources stop short of documenting a standard Configuration Management Database or Information Technology Infrastructure Library extension that automatically records exact agent-to-corpus bindings.

Assumptions

• Assumption: Teams can surface stable corpus or index version identifiers even when tools do not require them. Justification: aliases, index objects, and custom trace attributes already exist in the inspected infrastructure, so the missing piece is discipline rather than a missing technical hook.
• Assumption: For remote knowledge sources, an equivalent control can use source snapshot identifiers, retrieval-response identifiers, or timestamped export bundles when no local search index exists. Justification: Azure AI Search explicitly supports remote knowledge sources in agentic retrieval, so version pinning cannot rely only on local index names.
• Assumption: The inaccessible full ITIL practice guide would not reverse the governance mapping here, because the recommendation is limited to controls already visible in accessible rollout and content-governance sources: approval, impact review, staged promotion, and rollback readiness.

Analysis

The evidence was weighted toward foundational RAG papers and current platform documentation because the core question is operational causality, namely how a document change reaches inference-time behavior, rather than market positioning or vendor rhetoric.

Mechanism evidence is stronger than incident evidence: Lewis and Azure show how corpus changes alter the prompt surface, while Azure and ServiceNow show concrete ways stale, deleted, duplicate, or weakly structured content can survive into retrieval and summarization.

The evaluation frameworks are useful but incomplete for governance because they measure outcomes and traces, not authoritative dependency registration; prior repository work on dependency mapping and runtime divergence fills that missing control-plane perspective.

A rival response would be to keep corpora fully live for freshness and accept occasional regression, but the cited alias, blue-green, and reliability sources show that search infrastructure already exposes rollout and recovery primitives, so adopting unmanaged freshness remains a governance choice rather than a purely technical constraint.

Risks, Gaps, and Uncertainties

• Publicly accessible named postmortems for document-drift incidents remain sparse in the inspected source base, so incident prevalence and typical blast radius are lower-confidence than the mechanism and mitigation claims above.
• ServiceNow official documentation on Change Management, Dependency Views, and article versioning was not directly readable from the seeded Uniform Resource Locators (URLs) in this session, so ServiceNow-specific governance conclusions rely more on accessible community guidance and prior completed repository items than on first-party manual text.
• The evidence base is stronger for indexed corpora than for fully remote or ephemeral retrieval sources, so some version-pinning advice may need adaptation when the agent retrieves directly from live application programming interfaces (APIs) or remote knowledge sources.
• The inspected evaluation tools show how to compare runs and monitor retrieval quality, but they do not by themselves prove that most teams in production already run corpus-version-aware gates, so maturity of real-world adoption remains uncertain.

Open Questions

• Which minimum metadata set is sufficient for agent-to-corpus dependency registration: corpus identifier only, index alias plus timestamp, or retrieved-document hashes per run?
• What threshold of semantic delta or affected-article count should trigger mandatory regression testing before a corpus update is promoted?
• How should teams govern remote knowledge sources, where freshness is high but rollback and reproducibility are weaker than with pinned local indexes?

sources

• [x] Lewis et al. (2020) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• [x] Gao et al. (2024) Retrieval-Augmented Generation for Large Language Models: A Survey
• [x] Es et al. (2023) RAGAS: Automated Evaluation of Retrieval Augmented Generation
• [x] Ragas Documentation
• [x] Ragas RAG Evaluation Quickstart
• [x] Ragas Production Monitoring
• [x] LangSmith Evaluation Documentation
• [x] TruLens Instrumentation Overview
• [x] TruLens Groundtruth Evaluations for Retrieval Systems
• [x] Microsoft Foundry Retrieval Augmented Generation
• [x] Azure AI Search Overview
• [x] Azure AI Search Agentic Retrieval Overview
• [x] Azure AI Search Changed and Deleted Blob Handling
• [x] Azure AI Search Reliability
• [x] Azure AI Search Data Plane Representational State Transfer (REST) Application Programming Interfaces
• [x] Elasticsearch Aliases
• [x] Amazon OpenSearch Service Configuration Changes
• [x] DataStax RAGStack Indexing Guidance
• [x] ServiceNow Community: Best practices to use your knowledge articles with Now Assist
• [x] TEKsystems Structuring Knowledge for Gen AI in ServiceNow
• [ ] Axelos ITIL 4 Foundation landing page
• [ ] ServiceNow Change Management documentation
• [ ] ServiceNow Dependency Views documentation
• [ ] ServiceNow Knowledge article versioning documentation
• [x] Mitchell (2026) Knowledge Graph in the live execution path of multi-step Large Language Model systems
• [x] Mitchell (2026) Is knowledge scaffolding an established concept within context engineering for Large Language Models and AI agents?
• [x] Mitchell (2026) Dependency Mapping Across .NET Codebases, Terraform, Dynatrace, Confluence, Log Aggregation, and the Configuration and Service Data Model
• [x] Mitchell (2026) Declared versus runtime AIBOM divergence
• [x] Mitchell (2026) ServiceNow AI: Knowledge Management, RAG Pipelines, and Agent Frameworks
• [x] Mitchell (2026) ServiceNow orchestration and agentic AI roadmap