What are the primary failure modes in enterprise Artificial Intelligence (AI)…

Research Question

What are the primary failure modes in enterprise Artificial Intelligence (AI) and low-code deployments, including data leakage, conflicting automations, unintended actions by AI agents, and loss of auditability, and how can governance systems be designed with preventative and corrective controls that address each identified failure scenario?

Findings

Executive Summary

• The primary enterprise failure modes in Artificial Intelligence (AI) and low-code deployments are boundary failures, specifically untrusted content crossing into prompts or memory, over-broad action authority, silent wrong outputs, uncontrolled maker publication, and missing end-to-end audit trails, but their frequency and severity are amplified by broader delivery immaturity and systems-capability debt.
• Artificial Intelligence (AI)-specific failures are concentrated in prompt injection, retrieval contamination, and persistent memory poisoning, while low-code-specific failures are concentrated in citizen-development sprawl, connector sprawl, fragile promotion, and conflicting automations.
• Preventative controls should therefore narrow who can build, what can connect, where work can run, and how changes can promote, using managed environments, scoped connectors, authenticated channels, least privilege, and sequential release gates.
• Corrective controls must assume prevention will miss material cases, so runtime monitoring, immutable logs, quarantine, rollback, rate limits, memory reset, and practiced incident response are core governance components rather than optional add-ons.

Key Findings

1. High confidence: Prompt injection is a primary enterprise failure mode because untrusted external content can be reinterpreted as instructions, causing data exfiltration, unauthorized tool use, and manipulated downstream decisions in real-world Large Language Model (LLM) applications.
2. High confidence: Unintended actions become materially dangerous only when agents are granted excessive authority, so the decisive governance control is not better prompting alone but least privilege, narrowed tool surfaces, authenticated identities, and human approval for high-impact actions.
3. Medium confidence: Persistent memory and asynchronous agent workflows create a distinctive Artificial Intelligence (AI) failure class in which poisoned state can survive beyond the triggering session, so governance must include memory scoping, memory reset, and post-incident state cleanup.
4. High confidence: Silent but plausible wrong answers are governance failures rather than mere quality issues when organizations let operational or customer decisions depend on model output without verification, because liability and instability arrive before explicit technical alarms do.
5. High confidence: Low-code estates fail through shadow Information Technology (IT), technical debt, and conflicting automation when citizen developers can build and publish without centralized repositories, expert review, role clarity, and separate development, test, and production environments, and that pattern is reinforced when systems-capability debt keeps generating workaround demand.
6. High confidence: A consistently supported preventative control pattern is managed isolation plus staged promotion, specifically controlled maker environments, scoped connectors and channels, prevalidated deployments, and approval-based release gates, because those controls jointly remove the easiest unmanaged path from authoring to production.
7. High confidence: Loss of auditability is a primary failure mode because enterprises cannot reconstruct accountability without one evidence chain spanning prompts, knowledge sources, identities, approvals, runtime activity, and deployment artifacts.
8. Medium confidence: Detection gaps remain material because prompt injection is not deterministically solvable today, access failures can compound downstream effects, and low-code policy enforcement can lag across a large tenant, so circuit breakers, quarantine, rollback, rate limiting, and rehearsed incident response are indispensable corrective controls.
9. High confidence: The shared governance pattern across Artificial Intelligence (AI) and low-code systems is that failures concentrate at control-boundary crossings, while recurring severity is amplified by delivery immaturity, systems-capability debt, identity-model failure, access-control amplification, and information-architecture incoherence; the main Artificial Intelligence (AI)-specific additions are instruction ambiguity and persistent state and the main low-code-specific additions are maker sprawl and release fragility.

Assumptions

• Assumption: Robotic process automation governance lessons transfer materially to low-code application and agent programs. Justification: both shift automation authoring toward non-specialists while relying on central platform controls for safe release and support.
• Assumption: Publicly documented prompt injection patterns on general LLM applications are representative of the same risk class in enterprise internal assistants. Justification: the cited sources describe the vulnerability as architectural and content-path dependent, not consumer-product specific.

Analysis

• The most convincing evidence for Artificial Intelligence (AI)-specific failure is the combination of academic attack papers and Microsoft's production guidance, because together they show both feasibility and practical impact.
• The low-code evidence is less about spectacular exploits and more about repeated organizational failure, which is exactly what governance needs; across academic, public-sector, and enterprise platform guidance, the same preventive pattern repeats: isolate build spaces, centralize repositories, enforce review, and separate promotion from authoring.
• That consistency matters because it means governance is not merely aspirational: the platform surfaces already exist to narrow knowledge sources, channels, triggers, maker environments, and release paths, so the remaining problem is disciplined design and operating-model enforcement.
• The evidence also weighs against a purely preventative mindset, because silent failure, incomplete prompt-injection prevention, and post-hoc safeguard layers all point to the same conclusion: enterprises need reversible workflows, emergency stop paths, incident playbooks, and state cleanup because some failures will only be visible after an output or action has already occurred.

Risks, Gaps, and Uncertainties

• Public evidence for memory-poisoning incidents is still thinner than public evidence for prompt injection generally, so the memory-governance recommendations are based on a high-quality proof of concept plus architectural reasoning rather than on a large public incident corpus.
• Vendor documentation confirms enforcement latency and runtime quarantine behavior, but it does not quantify how often material harm occurs before enforcement completes, so the size of this exposure window is still uncertain.

Open Questions

• Which public platform patterns are most effective for memory sanitization and trusted-state reconstruction after agent compromise?
• How should enterprises quantify acceptable enforcement lag and rollback time across large low-code estates with mixed maker maturity?
• What minimum monitoring and approval set is sufficient for medium-risk write-capable agents before the control burden outweighs the productivity gain?

sources

• [x] AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) Repository — - accessible landing page for a public incident repository; used only as evidence that public incident collections exist because granular entries were not required for downstream claims.
• [x] AIAAIC harms taxonomy project — - public description of AIAAIC's evidence-based harms taxonomy work.
• [x] National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework (AI RMF) 1.0 publication page — - canonical framework publication page.
• [x] NIST AI RMF Core — - authoritative govern and map outcomes used for control mapping.
• [x] NIST AI RMF Playbook landing page — - current official landing page for the playbook.
• [x] NIST AI RMF Playbook - Govern — - suggested actions for policies, monitoring, incident response, and change management.
• [x] NIST AI RMF Playbook - Manage — - public references for incident databases, resilience resources, and contingency thinking.
• [x] MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS) home page — - checked, but the site rendered only a JavaScript shell in this runtime and was not used for downstream factual claims.
• [x] Gartner enterprise AI risk document — - checked, but remained analyst-access gated in this runtime and was not used for downstream factual claims.
• [x] Anthropic Responsible Scaling Policy — - frontier developer view of layered safeguards, asynchronous monitoring, and rapid jailbreak response.
• [x] Open Worldwide Application Security Project (OWASP) Top 10 for Large Language Model (LLM) Applications project — - checked as the project landing page.
• [x] Open Worldwide Application Security Project (OWASP) LLM01 Prompt Injection — - current detailed prompt injection taxonomy and mitigation guidance.
• [x] Open Worldwide Application Security Project (OWASP) Prompt Injection overview — - prompt injection overview with incident examples and mitigations.
• [x] Universal and Transferable Adversarial Attacks on Aligned Language Models — - evidence that jailbreak-style attacks can transfer across models.
• [x] Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — - core indirect prompt injection paper demonstrating real-world system exploitation.
• [x] How Microsoft defends against indirect prompt injection attacks — - vendor documentation of observed impacts and mitigations.
• [x] When AI Remembers Too Much: Persistent Behaviors in Agents' Memory — - proof of concept showing prompt injection poisoning of long-term agent memory.
• [x] Air Canada ordered to compensate customer misled by chatbot — - authoritative coverage of a production silent-failure liability case.
• [x] Governing Citizen Development to Address Low-Code Platform Challenges — - interview-based academic evidence on low-code governance debt, shadow Information Technology (IT), and technical debt.
• [x] A framework for implementing robotic process automation projects — - peer-reviewed evidence on robotic process automation implementation failure and socio-technical governance needs.
• [x] Digital.gov - 5 tips for implementing citizen development in your robotic process automation program — - public-sector guidance on governance, role-based access control, and environment separation for citizen development.
• [x] Copilot Studio security and governance — - Microsoft control surface overview for agent governance.
• [x] Copilot Studio data loss prevention — - granular low-code agent restrictions for authentication, knowledge, tools, channels, triggers, and Hypertext Transfer Protocol (HTTP) requests.
• [x] Power Platform data loss prevention overview — - design-time and runtime enforcement behavior, including suspension, quarantine, disabled connections, and enforcement latency.
• [x] Managed Environments overview — - baseline managed-environment control capability.
• [x] Managed governance in Power Platform — - environment groups, delegated admin, inventory, actions, and recommendations.
• [x] Environment routing — - maker isolation into managed developer environments with preconfigured governance settings.
• [x] Pipelines in Power Platform — - sequential promotion, approval, prevalidation, audit, and rollback controls.
• [x] 2025 DevOps Research and Assessment (DORA) report overview — - evidence that AI amplifies existing workflow and control weaknesses.
• [x] DevOps Research and Assessment (DORA) AI Capabilities Model report landing page — - evidence that platforms and platform teams are core scaling mechanisms.