How can a runtime-observed Artificial Intelligence Bill of Materials (AIBOM) be…

Research Question

How can a dynamic, runtime-observed Artificial Intelligence Bill of Materials (AIBOM) be generated for an agentic Artificial Intelligence (AI) system, capturing execution traces, transient Retrieval-Augmented Generation (RAG) retrievals, tool call outputs, and memory state at decision points, and what formal models describe the divergence between a declared design-time AIBOM and what actually executes at inference time?

Findings

Executive Summary

A runtime-observed AIBOM is best generated as an event-sourced provenance graph plus decision-point state snapshots, and divergence should be expected whenever runtime retrieval, delegated authority, mutable external state, or dynamic orchestration choices shape the run. The strongest single-run ground truth is the causality graph of entities, activities, and agents linked by trace correlation and backed by immutable runtime events, while snapshots preserve transient context such as retrieved chunks, prompt windows, memory state, and delegated authority. Declared-to-observed divergence should be modelled explicitly across coverage, configuration, retrieval, memory, authority, topology, external-state, and control-policy surfaces rather than treated as a generic drift label. Replayability is partial: evidence-preserving reconstruction is often achievable, but behaviour-reproducing re-execution is inherently limited by stochastic generation, mutable external systems, and evolving retrieval corpora.

Key Findings

1. A usable runtime AIBOM needs a layered representation that records trace topology, content-bearing runtime events, and decision-point state snapshots, because none of those layers alone is sufficient to reconstruct one agent run faithfully.
2. The best formal ground truth for a single non-deterministic run is a full causality graph backed by immutable events, while snapshots and statistical summaries are derived projections for transient-state preservation and cross-run governance analysis respectively.
3. Declared-versus-observed divergence is structurally multi-dimensional, because observed runs can differ in component coverage, model configuration, retrieval set, memory state, caller authority, orchestration path, external world state, and active logging or guardrail policy.
4. Runtime AIBOM generation should treat OpenTelemetry and W3C Trace Context as the correlation substrate, W3C PROV as the provenance vocabulary, and event sourcing as the persistence pattern for the immutable run record.
5. Deterministic replay is realistic for captured prompts, tool arguments, recorded tool outputs, and policy decisions, but live re-execution of model output or external API results remains only partially reproducible even when seeds and parameters are logged.
6. Authority information must be part of the runtime AIBOM itself rather than inferred later, because delegated identity chains and effective permissions are not fully preserved by runtime token formats or trace correlation alone.
7. For governance purposes, the most consequential runtime AIBOM divergences are not generic variance but the ones that change what information, authority, or control setting shaped the decision, because those are the divergences that alter risk exposure and auditability.

Assumptions

• Assumption: The unmatched seeded provenance-paper titles do not overturn the core model here. Justification: the conceptual conclusions rely mainly on accessible standards that already define provenance, trace correlation, and replay-oriented event history.
• Assumption: Memory state can be treated as a snapshot surface rather than as a continuously logged stream. Justification: the governance question is what state informed a decision point rather than every intermediate token-level mutation.

Analysis

The evidence supports modelling the runtime AIBOM as an immutable run-history object rather than as a mutable inventory record, because the strongest sources all describe provenance, tracing, and event history in terms of ordered activities and correlated state changes. OpenTelemetry contributes the practical correlation and field vocabulary, but its Generative AI semantic conventions are still in development, so it is a useful substrate rather than a complete stable specification for the whole runtime AIBOM. AWS Bedrock is useful because it demonstrates that production agent traces already include the run objects this theory needs, including caller chain, prompt text, orchestration step types, rationale, observations, and failure traces, and it supports the inference that runtime evidence remains fragmented across multiple log modes. The declared design-time AIBOM and the runtime AIBOM should therefore be treated as linked but non-identical artifacts: the first declares allowed capability and expected structure, while the second records an actual realized path through that design space. A practical governance program should explicitly reject the rival idea that one can preserve per-run truth using only summary metrics or only a final-state snapshot, because both approaches lose causal structure and make later incident reconstruction materially weaker.

Risks, Gaps, and Uncertainties

The OpenTelemetry Generative AI conventions are still marked development status, so a runtime AIBOM built directly on those field names may need later schema adaptation. The unmatched seeded provenance-paper titles leave the academic framing thinner than ideal, even though the standards and platform sources are sufficient to support the core conceptual model presented here. The boundary between "memory state snapshot" and "continuous memory provenance" remains implementation-sensitive, so some systems may need deeper state capture than this conceptual baseline assumes.

Open Questions

• What minimum snapshot schema is sufficient to preserve agent memory state at a decision point without storing every intermediate memory mutation?
• How should a declared AIBOM express acceptable Retrieval-Augmented Generation (RAG) variance so that normal retrieval behaviour is not misclassified as harmful divergence?
• Which divergence classes should trigger immediate policy intervention versus post-run audit only?

sources

• [x] OpenTelemetry Generative AI Semantic Conventions - overall standard surface for Generative Artificial Intelligence (AI) telemetry
• [x] OpenTelemetry Traces Concept - trace, span, event, link, and correlation model
• [x] OpenTelemetry Generative AI Events - request, response, seed, token, tool, and conversation event fields
• [x] OpenTelemetry Generative AI Agent Spans - agent and tool execution span model
• [x] OpenTelemetry Project - vendor-neutral observability substrate
• [x] World Wide Web Consortium (W3C) Trace Context Recommendation - standard trace propagation model
• [x] World Wide Web Consortium (W3C) PROV Overview - provenance concepts, constraints, and reproducibility goals
• [x] Amazon Web Services (AWS) Bedrock Agent Trace Events - concrete runtime agent trace schema
• [x] Amazon Web Services (AWS) Bedrock Model Invocation Logging - request, response, metadata, and log-destination behaviour
• [x] Fowler (2005) Event Sourcing - event-log, replay, temporal-query, and snapshot pattern
• [x] National Institute of Standards and Technology (NIST) AI Risk Management Framework - lifecycle governance context for runtime evidence
• [x] David Mitchell (2026) AI low-code observability telemetry governance - completed repository item on governance telemetry surfaces
• [x] David Mitchell (2026) AIBOM identity delegation trust theory - completed repository item on delegation-chain persistence and runtime authority
• [x] David Mitchell (2026) UELGF runtime monitoring for non-deterministic behaviour - adjacent completed repository item on runtime precursor signals for stochastic agent systems