Compliance Risks of Relying on Stochastic Large Language Model (LLM) Outputs…

Research Question

What evidence or guidance exists on the compliance risks of relying primarily on stochastic Large Language Model (LLM) outputs for governance, privacy, or regulatory decisions?

Findings

Executive Summary

Relying primarily on probabilistic and potentially variable Large Language Model outputs for governance, privacy, or regulatory decisions creates a material compliance risk because the strongest accessible guidance requires contestable, accountable, and reviewable decision processes, while empirical evidence shows that Large Language Models can vary, hallucinate, and obscure traceability. Financial-services and cross-sector risk-management guidance reinforce the same direction by emphasizing governance, monitoring, validation, clear roles, and executive responsibility rather than permitting unbounded reliance on generative outputs. The FTC's DoNotPay action already shows that at least one regulator will challenge AI systems marketed as substitutes for legal expertise or automated legal-compliance checking when testing and evidence are missing. The best-supported mitigation is a hybrid pattern in which the model proposes, summarizes, or prioritizes, while deterministic rules, auditable policy checkpoints, and meaningful human escalation make the final consequential decision.

Key Findings

1. European Union and United Kingdom data-protection guidance restricts solely automated decisions with legal or similarly significant effects and requires meaningful safeguards, so probabilistic and potentially variable Large Language Model outputs are a weak choice for sole authority in privacy or governance decisions.
2. Meaningful human intervention must be able to understand, challenge, and change an automated outcome, which means nominal review layered on top of a probabilistic model does not remove the compliance risk if the reviewer is only rubber-stamping.
3. United Kingdom financial regulators and NIST frame Artificial Intelligence governance as a problem of accountability, monitoring, validation, and role clarity, which supports controlled use of models but not blind reliance on probabilistic outputs for final compliance judgments.
4. Current United States banking model-risk guidance excludes generative and agentic Artificial Intelligence from scope while preserving governance, validation, monitoring, and third-party oversight expectations for covered models.
5. The Federal Trade Commission's DoNotPay action shows that the Commission will challenge claims that a chatbot can substitute for professional legal services or automated legal-compliance checking when testing and evidence are missing.
6. Official legal-practice guidance and clinical literature both show that Large Language Models can fabricate authorities, vary across repeated prompts, and provide unsafe or untraceable recommendations, which makes them a poor fit for compliance workflows that depend on accuracy, consistency, and auditable reasoning.
7. Foundation-model research indicates that misinformation, privacy leakage, and automation harms are structural downstream risks, so using one stochastic model across many governance tasks concentrates rather than localizes compliance exposure.
8. The strongest currently supported design is a hybrid architecture in which the Large Language Model prepares proposals or summaries while deterministic policies, audit logs, and empowered human escalation retain final decision authority for consequential actions.

Assumptions

• This item treats privacy classifications, access approvals, compliance escalations, and regulatory reporting judgments as governance decisions that can become legally or operationally significant even when they do not map exactly to Article 22 cases, because the safeguard logic still informs defensible design.
• Exclusion of generative and agentic Artificial Intelligence from current banking model-risk guidance is treated here as a reason to tighten local governance rather than as permission to weaken controls, because the official sources emphasize broader risk-management responsibility.

Analysis

The evidence weighs most heavily toward regulator and framework sources that describe what a controlled decision process must contain: safeguards, accountability, monitoring, and human authority. That weighting matters because the core compliance problem is not only whether the model is accurate on average, but whether a firm can justify the individual decision path when a regulator, auditor, or affected person asks for explanation and correction. The legal and clinical reliability sources then supply the operational reason not to trust stochastic output as final authority: the system can produce convincing but false authorities, inconsistent answers, and opaque source chains even when the prose looks authoritative. One plausible rival remedy is to rely mainly on better models or prompt engineering, but the reviewed regulator sources still ask for governance structures outside the model, and the reviewed empirical sources do not show that prompt quality removes hallucination, variability, or traceability risk completely. Another plausible rival remedy is blanket human review of every case, but privacy guidance requires meaningful intervention rather than symbolic approval, and blanket queues can still fail if reviewers lack time, evidence, or authority to change the model's output. The best-supported operating model is therefore to keep the Large Language Model where variance is tolerable, summarization, drafting, prioritization, and proposal generation, and move final allow, deny, classify, or report decisions into deterministic and reviewable control paths.

Risks, Gaps, and Uncertainties

• Public banking guidance is currently clearer about governance expectations than about detailed generative-AI validation standards, because the latest interagency model-risk guidance excludes generative and agentic systems from scope.
• Foundation-model risk literature is broad and not written as sector-specific compliance guidance, so it strengthens structural-risk claims more than it proves any one regulator's enforcement theory.

Open Questions

• Which published financial-services incidents most clearly connect stochastic model variance to audit or compliance breach, rather than to general model-risk concern?
• What minimum evidence package should a reviewer see before overturning or approving a model-generated governance recommendation in a high-volume workflow?
• Can a standard policy-decision schema be defined for privacy classification, access approval, and regulatory-report drafting so that the same deterministic controls can govern all three?

sources

• [x] Information Commissioner's Office Guidance on AI and Data Protection
• [x] Article 29 Working Party (2018) Guidelines on Automated individual decision-making and Profiling
• [x] European Commission Restrictions on Automated Decision-Making
• [x] Bank of England, Prudential Regulation Authority, and Financial Conduct Authority (2022) DP5/22 Artificial Intelligence and Machine Learning
• [x] Bank of England, Prudential Regulation Authority, and Financial Conduct Authority (2023) FS2/23 Artificial Intelligence and Machine Learning
• [x] Office of the Comptroller of the Currency, Federal Reserve, and Federal Deposit Insurance Corporation (2026) Model Risk Management Revised Guidance
• [x] National Institute of Standards and Technology Artificial Intelligence Risk Management Framework Core
• [x] Federal Trade Commission (2024) FTC Announces Crackdown on Deceptive AI Claims and Schemes
• [x] National Center for State Courts (2026) A Legal Practitioner's Guide to AI and Hallucinations
• [x] Weidinger et al. (2021) Ethical and social risks of harm from Language Models
• [x] Bommasani et al. (2022) On the Opportunities and Risks of Foundation Models
• [x] Roustan and Bastardot (2025) The Clinicians' Guide to Large Language Models: A General Perspective With a Focus on Hallucinations