What security capabilities are required in an enterprise Artificial…

Research Question

What security capabilities are required in an enterprise Artificial Intelligence (AI) system, beyond basic Application Programming Interface (API) access controls and audit logging, to address prompt injection (direct and indirect), Retrieval-Augmented Generation (RAG)-based attacks (data poisoning, context manipulation, indirect injection via retrieved documents), model supply chain compromise (malicious fine-tuned weights, compromised model registries, trojan base models), and data exfiltration (sensitive data leakage through model outputs or tool calls), and how should these capabilities be incorporated into a complete enterprise AI security threat model?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

Enterprise AI systems need a security stack that goes beyond access control and audit logs by adding controls for prompt and retrieval boundary integrity, model and connector provenance, identity-scoped execution, deterministic egress control, and runtime circuit breakers.

Prompt injection and Retrieval-Augmented Generation (RAG) attacks are structural because trusted instructions and untrusted content share model context, so the enterprise has to constrain what the model can do even after content is retrieved or interpreted.

Model supply chain compromise and unsafe model-loading paths are already operational risks, not hypothetical edge cases, because public machine-learning ecosystems have documented malicious dependency compromise and still rely heavily on formats that can execute code at load time.

The resulting enterprise threat model should treat the prompt plane, retrieval corpus, model artifact pipeline, tool-execution surface, and runtime governance plane as separate assets with distinct indicators and control families, then map those families back into the enterprise capability model as explicit security subdomains.

Key Findings

1. A baseline of Application Programming Interface access controls plus audit logging is insufficient for enterprise AI, because it does not verify prompt integrity, retrieval-boundary correctness, model provenance, deterministic tool mediation, or pre-action containment once a model can ingest untrusted content and invoke connected systems.
2. Prompt injection remains a structural risk rather than a filter-tuning problem, because authoritative sources and current research converge on the point that indirect content can still steer model behavior unless control flow, tool access, and approval boundaries are enforced outside the model itself.
3. Retrieval-Augmented Generation introduces a second security boundary beyond prompt injection, because poisoned documents, stale access-control metadata, embedding inversion, and retrieval-database membership leakage can all expose or reshape sensitive knowledge even when ordinary authentication is present.
4. Model supply chain security must be treated as a first-class enterprise capability, because compromised machine-learning dependencies and unsafe serialization formats have already created code-execution and exfiltration paths that ordinary access controls do not detect before model loading occurs.
5. Data exfiltration control for enterprise AI has to be enforced at the agent-execution layer, because the decisive risk is whether the system can read sensitive data and send it through tools, triggers, or outbound connectors before a human or audit process reviews the event.
6. The minimum additional control stack is separate machine identities, least-privilege delegation, prompt and document attack detection, permission-safe retrieval architecture, model-artifact provenance and scanning, deterministic egress controls, human approval for high-consequence actions, and runtime anomaly detection with halt or quarantine paths.
7. The evidence base is strongest for indirect prompt injection telemetry, retrieval-leakage research, and machine-learning dependency compromise, while public evidence for large-scale nation-state use of these exact AI-native techniques remains materially thinner than the evidence for researchers, red teams, and opportunistic attackers.
8. In the enterprise AI capability model, security should no longer be a single generic domain, because the evidence supports at least four distinct security subdomains, prompt and retrieval boundary defense, identity-scoped execution control, model and connector supply-chain assurance, and runtime assurance.

Assumptions

• The target enterprise wants AI systems that can call tools or process third-party documents rather than a strictly isolated chat interface. Justification: the control gap only becomes material once the model can read untrusted content or cause side effects.
• The enterprise either consumes public model artifacts directly or inherits upstream components built from public machine-learning ecosystems. Justification: otherwise model-supply-chain risk can be reduced substantially through full internal curation and signing.

Analysis

The complete enterprise threat model separates five assets: the prompt and instruction plane, the retrieval corpus and permission state, the model artifact and dependency pipeline, the tool-execution surface, and the runtime governance plane. Prompt injection primarily targets instruction integrity and tool authority; Retrieval-Augmented Generation attacks target retrieval integrity, permission correctness, and corpus confidentiality; supply-chain attacks target model provenance and loader safety; exfiltration attacks target secrets in context, outputs, and tool arguments; and runtime-governance failures target the organization's ability to detect unsafe precursor signals before action.

The control trade-off is between adaptability and determinism. Enterprises want agents to reason flexibly over new content and workflows, but the evidence shows security cannot be delegated to that same flexible reasoning loop. The durable pattern is therefore layered: scoped identity and policy outside the model, bounded retrieval inside explicit knowledge architectures, safe model promotion through provenance checks, and runtime monitoring that can halt execution when behavior moves outside the permitted envelope.

Relative to the "API access + audit" baseline, the decisive additions are not more logs but more chokepoints. The enterprise needs promotion-time checks before models, prompts, and tools reach production; execution-time checks before tools or outbound channels are used; and runtime checks that stop suspicious sequences before they accumulate into machine-speed harm. Audit logs remain necessary, but in this evidence set they are forensic evidence, not the main preventive control.

Risks, Gaps, and Uncertainties

• The prompt-injection evidence base now includes in-the-wild telemetry, but the public record still exposes only part of actual attacker prevalence and does not yet support a precise ranking of threat-actor classes.
• Retrieval-leakage evidence is strong in research settings, but the exact exploitability of every managed enterprise vector service will still depend on its exposure model, permission architecture, and outbound interface surface.
• Safer serialization and trust signals reduce model-loading risk, but they do not by themselves prove that a model is behaviorally safe, policy-compliant, or free of targeted backdoors.

Open Questions

1. What is the minimum practical evidence package for promoting a third-party model, adapter, or connector into a regulated enterprise environment?
2. Which runtime precursor signals are most predictive of exfiltration attempts before any data leaves the boundary?
3. How should enterprises quantify acceptable stale-permission windows for copied retrieval corpora?
4. Which evaluation suite best tests combined prompt, retrieval, and tool-path abuse in the same agent workflow?

sources

• [x] Mitchell (2026) Prompt injection threat landscape - foundational corpus item on prompt injection attacks and defences
• [x] Mitchell (2026) Permission-safe Retrieval-Augmented Generation (RAG) enterprise information architecture - prior corpus item on retrieval boundary correctness, access representation, and retrieval-store leakage
• [x] Mitchell (2026) AI agent identity and access management enterprise - prior corpus item on machine identity, delegation, and attribution
• [x] Mitchell (2026) AI agent control-plane architecture enterprise - prior corpus item on external policy enforcement and observability planes
• [x] Mitchell (2026) Access control amplification under agentic operations - prior corpus item on machine-speed privilege amplification
• [x] Mitchell (2026) UELGF agentic AI specific risks and runtime monitoring - prior corpus item on runtime precursor monitoring and circuit breakers
• [x] OWASP GenAI LLM01 Prompt Injection - authoritative taxonomy for prompt injection, agency-sensitive impact, and baseline mitigations
• [x] OWASP Top 10 for Large Language Model Applications - canonical list showing prompt injection, supply chain, sensitive information disclosure, and excessive agency as distinct risks
• [x] MITRE ATLAS - authoritative AI-adversary taxonomy baseline
• [x] NIST AI Risk Management Framework (AI RMF) Core - lifecycle risk-management and third-party governance expectations
• [x] Greshake et al. (2023) Not what you've signed up for - foundational indirect prompt injection paper for real-world Large Language Model-integrated applications
• [x] Debenedetti et al. (2025) Defeating Prompt Injections by Design - capability-based defence showing explicit control/data-flow separation
• [x] Palo Alto Networks Unit 42 (2026) AI agent prompt injection telemetry - in-the-wild observations of indirect prompt injection payloads and attacker intents
• [x] Microsoft Azure AI Content Safety Prompt Shields - user-prompt and document-attack detection controls
• [x] Morris et al. (2023) Text Embeddings Reveal (Almost) As Much As Text - embedding inversion evidence
• [x] Anderson et al. (2024) Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation - retrieval-database membership leakage evidence
• [x] PyTorch (2022) Compromised nightly dependency - official real-world machine-learning supply-chain compromise case
• [x] Hugging Face Docs Pickle security - official documentation on unsafe model serialization, signed commits, and trust boundaries
• [x] Hugging Face (2023) Safetensors security audit - external audit result for safer tensor serialization
• [x] AWS Security Blog (2026) Four security principles for agentic AI systems - explicit guidance on machine-speed privilege amplification and deterministic external controls