How do you capture a runtime-observed Artificial Intelligence Bill of Materials…

Research Question

How do you instrument a real agentic Artificial Intelligence workload, meaning a tool-using workload that plans or acts across multiple steps, to capture a runtime-observed Artificial Intelligence Bill of Materials (AIBOM), specifically using OpenTelemetry (OTel) semantic conventions for Generative Artificial Intelligence and platform-native observability tools, Amazon Web Services (AWS) Bedrock AgentCore Observability and LangSmith, and what does the captured trace data reveal about divergence from the declared AIBOM constructed in 2026-05-06-aibom-declared-construction-practice?

Findings

Executive Summary

Runtime-observed AIBOM capture is operationally feasible today for both AWS Bedrock and LangGraph, but only as a layered telemetry assembly rather than as a single native export.

Bedrock provides the stronger native substrate because AgentCore and Bedrock trace events already expose session and trace structure, prompts, inference settings, tool or knowledge-base activity, and caller chains, while LangGraph usually reaches the same runtime AIBOM fidelity only when LangSmith is combined with custom OpenTelemetry spans or framework auto-instrumentation.

The runtime trace surface consistently diverges from the declared AIBOM by adding execution-specific state such as session identifiers, actual tool-call order, retrieved context, token usage, and failure paths, while some declared but dormant components remain unseen in any single run.

A practical storage design therefore needs a collector-mediated pipeline plus backend-specific retention and query strategy, because the runtime AIBOM is only as useful as the ability to preserve, search, and correlate the traces that instantiate it.

Key Findings

1. AWS Bedrock can populate runtime AIBOM fields for session identity, trace hierarchy, prompt text, inference configuration, tool or knowledge-base activity, and caller chains, which makes its native telemetry unusually close to the topology and content layers required by the runtime AIBOM model.
2. LangGraph can also yield a usable runtime AIBOM when the operator composes LangSmith traces with OpenTelemetry spans and optional framework instrumentation, because default run trees do not automatically capture every governance-relevant state surface such as memory snapshots or delegated authority context.
3. OpenTelemetry's Generative Artificial Intelligence events and agent spans already offer a workable field vocabulary for runtime AIBOMs, including models, messages, tools, tokens, agent identifiers, and versions, but decision-state fields such as checkpointed memory or effective permissions still require application-specific attributes or linked artifacts.
4. The runtime trace surface necessarily diverges from the declared AIBOM because one observed execution reveals actual session identifiers, tool-call order, retrieved context, token usage, and failure paths that a design-time inventory cannot fully specify in advance.
5. A single run can legitimately omit declared components such as unused tools, untaken graph branches, or unqueried knowledge bases, so absent-component divergence must be distinguished from real configuration drift before a runtime AIBOM is used as a compliance or incident-review artifact.
6. Missing-observability divergence is a first-class governance risk because runtime AIBOM completeness depends on enabling transaction search, content capture, and custom span emission, which means an apparently well-instrumented system can still hide decisive execution detail if the telemetry path is only partially configured.
7. The OpenTelemetry Collector is a practical ingress point for runtime AIBOM pipelines because it can batch, retry, redact, and fan out traces before they reach storage backends, allowing one instrumented workload to support both developer tooling and long-retention governance stores.
8. Backend choice determines how searchable and durable a runtime AIBOM becomes, with LangSmith optimizing for developer runs, Tempo optimizing for low-cost high-volume trace retention, and Jaeger or OpenSearch providing more explicit general-purpose trace-query and archive patterns for operational investigations.

Assumptions

• Assumption: The representative runtime AIBOM document uses placeholders because the task scope is documented practice rather than a live tenant walkthrough. Justification: The official Bedrock and LangSmith setup guides are sufficient to specify capture mechanics, but they do not provide one shared public trace payload for this exact comparison.
• Assumption: Current OpenTelemetry field names are suitable as the baseline runtime vocabulary even though the Generative Artificial Intelligence semantic conventions are not yet stable. Justification: The conventions are already detailed enough to model prompts, tools, usage, and agent fields, which is the practical requirement for this item.
• Assumption: LangGraph teams that need governance-grade runtime AIBOM capture will accept custom span emission or metadata attachment rather than relying on default framework traces alone. Justification: The official LangSmith, Phoenix, and OpenTelemetry materials all present manual or semi-automatic instrumentation as the mechanism for richer trace semantics.

Analysis

The evidence supports treating runtime AIBOM capture as an observability-architecture problem instead of only a schema problem, because the decisive question is not whether fields can be named, but whether they are emitted, retained, and queryable in one correlated trace path.

Bedrock currently offers the lower-friction implementation path for native runtime evidence, because the platform already couples agent execution with trace events and CloudWatch views, whereas LangGraph gives a more open but more operator-dependent stack that must be assembled deliberately.

The declared-versus-observed comparison becomes most useful when treated as a divergence classifier rather than a pass-fail diff, because some gaps are expected properties of single-run evidence while others indicate observability design failure or genuine runtime drift.

The storage evidence favors a two-plane design in practice, with one plane optimized for engineering investigation and another optimized for durable governance queries or archives, because trace backends differ materially in retention posture, query language, and operational cost.

The main unresolved weakness is decision-state capture, because prompt, tool, and model activity are now well-covered by current tooling while effective authority, exact memory state, and some policy outcomes still depend on application-specific instrumentation design.

Risks, Gaps, and Uncertainties

• This item is grounded in official documentation rather than a live tenant execution, so it demonstrates what can be captured and how to wire it, not an empirically observed production trace from one named agent.
• OpenTelemetry's Generative Artificial Intelligence semantic conventions are still in development status, so field names and maturity assumptions may change after this item's completion.
• Bedrock service-provided observability for runtime-hosted agents emphasizes default metrics, which means organizations can still end up with incomplete runtime AIBOM traces if they do not add custom agent spans or enable the required CloudWatch trace path.
• LangSmith's retention and developer-oriented run model are useful for debugging and evaluation, but long-term governance archives may still require a separate backend under operator-controlled retention policy.
• Content-bearing traces can improve runtime AIBOM completeness while simultaneously increasing the sensitivity of the stored trace corpus, so trace-capture scope must be aligned with data-governance policy rather than enabled indiscriminately.

Open Questions

• What minimum custom span schema is sufficient to make memory snapshots and effective authority portable across Bedrock, LangGraph, and other agent runtimes?
• When a runtime AIBOM is used for incident response, which fields should be stored directly in traces and which should be linked to colder content stores or checkpoint artifacts?
• Can a standards-aligned runtime AIBOM profile be defined on top of current OpenTelemetry semantic conventions without fragmenting as the conventions evolve?

sources

• [x] OpenTelemetry Semantic Conventions for Generative AI - canonical status page for the Generative Artificial Intelligence semantic conventions and their maturity level.
• [x] OpenTelemetry Generative AI Events - event-level fields for prompts, outputs, tools, system instructions, tokens, and seeds.
• [x] OpenTelemetry Generative AI Agent Spans - agent and tool span model for agent identifiers, versions, providers, and tool execution.
• [x] OpenTelemetry AWS Bedrock Semantic Conventions - AWS Bedrock-specific fields such as guardrail and knowledge-base identifiers.
• [x] OpenTelemetry Python Getting Started - official Python instrumentation guidance for opentelemetry-instrument, automatic instrumentation, and manual child spans.
• [x] OpenTelemetry Collector - official guidance on collector roles, batching, retries, filtering, and backend fan-out.
• [x] Amazon Bedrock AgentCore Observability - official overview of AgentCore observability and its OpenTelemetry-compatible telemetry.
• [x] Amazon Bedrock AgentCore Observability Get Started - official setup steps for runtime-hosted and non-runtime-hosted agents, including AWS Distro for OpenTelemetry (ADOT).
• [x] Amazon Bedrock AgentCore Telemetry Concepts - official definitions of sessions, traces, spans, and their hierarchy for agent monitoring.
• [x] Amazon Bedrock AgentCore Observability Views - official CloudWatch log-group, trace, span, and metrics viewing guidance.
• [x] Amazon Bedrock Agent Trace Events - official Bedrock per-step trace fields for prompts, rationale, action invocations, observations, and caller chains.
• [x] LangSmith Trace with OpenTelemetry - official LangSmith OpenTelemetry tracing guide and attribute mapping.
• [x] LangSmith Observability Concepts - official definitions of projects, traces, runs, threads, and retention behavior.
• [x] Traceloop OpenLLMetry Introduction - official overview of OpenLLMetry as OpenTelemetry-based observability for model applications.
• [x] OpenTelemetry Instrumentation for LangChain - official package page for LangChain auto-instrumentation behavior and privacy defaults.
• [x] Arize Phoenix Tracing Instrumentation - official Phoenix tracing patterns for manual spans and OpenInference attribute helpers.
• [x] Grafana Tempo - official overview of Tempo as a high-scale tracing backend using object storage.
• [x] Grafana Tempo TraceQL - official query model for selecting traces and deriving metrics from traces.
• [x] OpenSearch Trace Analytics - official OpenSearch trace analytics overview for OpenTelemetry ingestion and visualization.
• [x] Jaeger Storage - official Jaeger storage backend guidance, including OpenSearch and archive storage.
• [x] Jaeger Architecture - official Jaeger role and pipeline guidance, including placing the OpenTelemetry Collector in front of Jaeger.
• [x] David Mitchell (2026) Runtime AIBOM generation divergence theory - prior repository item defining the runtime AIBOM model used here.
• [x] David Mitchell (2026) Declared AIBOM construction practice - prior repository item defining the declared AIBOM surfaces compared here.
• [x] David Mitchell (2026) AI low-code observability telemetry governance - prior repository item on multi-stream governance evidence and telemetry correlation.
• [x] David Mitchell (2026) AIBOM platform observability control comparison - prior repository comparison of Bedrock and other platform control surfaces.
• [x] David Mitchell (2026) AIBOM identity attribution multiagent practice - adjacent practice item on runtime identity and delegation fields.