Explainable Artificial Intelligence (XAI)

Explainable Artificial Intelligence (XAI): current research state, leading institutions, and regulatory intersection in heavily regulated industries

2026-04-30 · governance-policy security-risk mlops-deployment agentic-ai · medium · source → · wiki →

key claims

The current XAI field is organised in the literature as a taxonomy of complementary explanation methods, with stable axes such as local versus global, ante-hoc versus post-hoc, and model-specific versus model-agnostic appearing across the survey literatureArrieta et al. (2020)Kim (2017)
Public research leadership in XAI is distributed across agenda-setting programmes and standards bodies such as DARPA and NIST, while widely reused methods such as LIME, SHAP, and TCAV came from different author groups rather than one frontier institutionDefense (n.d.)Phillips et al. (2021)Ribeiro et al. (2016)Lee (2017)Kim et al. (2018)
Named XAI techniques explain different things, because LIME provides local surrogate explanations, SHAP assigns local feature contributions, and TCAV maps internal behaviour to human conceptsRibeiro et al. (2016)Lee (2017)Kim et al. (2018)
Data-protection rules do not require source-code disclosure, but they do require meaningful information, significance, consequences, and contestability when solely automated decisions have legal or similarly significant effects on a personEuropean (n.d.)Information (n.d.)Article (n.d.)
The EU AI Act classifies creditworthiness, life and health insurance, and qualifying medical-device uses as high-risk and attaches logging, user information, human oversight, technical documentation, and robustness obligations to those systemsEuropean (n.d.)
Financial-services governance sources treat explainability as part of model-risk management and auditability, because SR 11-7, the Bank of England material, APRA CPS 230, and ISO/IEC 42001 all emphasise documentation, monitoring, effective challenge, critical operations, and traceable accountabilityFederal (n.d.)Bankofengland (n.d.)Bankofengland (n.d.)Bankofengland (n.d.)Australian (n.d.)International (2023)
In practice, XAI is most defensible in regulated audit and review when it is used as supporting evidence for validation, challenge, and review decisions, because no reviewed framework allows an explanation artifact to replace human accountability for a consequential decisionFederal (n.d.)Bankofengland (n.d.)AI (n.d.)
Agentic systems create a harder explanation problem than single-model prediction systems, because decision responsibility is spread across prompts, model calls, tool invocations, and handoffs, while current mechanistic-interpretability work still captures only a partial and labor-intensive view of internal computationAnthropic (n.d.)Phillips et al. (2021)European (n.d.)

Research Question

What is the current state of Explainable Artificial Intelligence (XAI) research, who leads it and what are the primary techniques, and how does XAI intersect with regulatory obligations, audit requirements, and accountability for automated decisions made by Artificial Intelligence (AI) agents in heavily regulated industries such as financial services and healthcare?

Findings

Executive Summary

Current regulation in heavily regulated industries requires explainability mainly as a governance capability, not as a mandate to use any single Explainable Artificial Intelligence (XAI) method. GDPR and Information Commissioner's Office (ICO) guidance still leaves room for cases where meaningful contestability requires a deeper account of model logic, but the cited texts frame that requirement in terms of meaningful information and safeguards rather than by naming a mandatory XAI method. The strongest current obligations converge on logging, technical documentation, meaningful user information, human intervention, monitoring, and named accountability. SHAP, Local Interpretable Model-agnostic Explanations (LIME), Testing with Concept Activation Vectors (TCAV), and related techniques are useful components of that governance stack, especially for validation, challenge, and audience-specific explanation artifacts, but none of them by itself satisfies the full regulated-sector burden. For agentic AI systems, current internal-mechanism research is promising but still too immature to replace workflow provenance, bounded authority, and human review.

Key Findings

The current XAI field is organised in the literature as a taxonomy of complementary explanation methods, with stable axes such as local versus global, ante-hoc versus post-hoc, and model-specific versus model-agnostic appearing across the survey literature.
Public research leadership in XAI is distributed across agenda-setting programmes and standards bodies such as DARPA and NIST, while widely reused methods such as LIME, SHAP, and TCAV came from different author groups rather than one frontier institution.
Named XAI techniques explain different things, because LIME provides local surrogate explanations, SHAP assigns local feature contributions, and TCAV maps internal behaviour to human concepts.
Data-protection rules do not require source-code disclosure, but they do require meaningful information, significance, consequences, and contestability when solely automated decisions have legal or similarly significant effects on a person.
The EU AI Act classifies creditworthiness, life and health insurance, and qualifying medical-device uses as high-risk and attaches logging, user information, human oversight, technical documentation, and robustness obligations to those systems.
Financial-services governance sources treat explainability as part of model-risk management and auditability, because SR 11-7, the Bank of England material, APRA CPS 230, and ISO/IEC 42001 all emphasise documentation, monitoring, effective challenge, critical operations, and traceable accountability.
In practice, XAI is most defensible in regulated audit and review when it is used as supporting evidence for validation, challenge, and review decisions, because no reviewed framework allows an explanation artifact to replace human accountability for a consequential decision.
Agentic systems create a harder explanation problem than single-model prediction systems, because decision responsibility is spread across prompts, model calls, tool invocations, and handoffs, while current mechanistic-interpretability work still captures only a partial and labor-intensive view of internal computation.

Assumptions

Assumption: The public ISO summary is sufficient for high-level governance claims, but not for clause-by-clause obligations. Justification: the accessible official ISO page describes management-system purpose, traceability, transparency, and continual improvement, but the full standard text is paywalled.
Assumption: The FDA draft-guidance landing page is adequate evidence for a current regulatory direction in healthcare, but not for a final binding obligation. Justification: the guidance is official and current, but still draft and not the sole basis of the healthcare conclusion.

Analysis

The key interpretive move is to separate explanation methods from explanation obligations. The methods literature asks how to make model behaviour more understandable, while the legal and supervisory material asks what an institution must disclose, document, review, monitor, and be accountable for. That distinction explains why regulators rarely name SHAP or LIME directly: they regulate the control objective, not the internal analytics implementation.

The competing GDPR interpretation is that "meaningful information about the logic involved" can require a more substantive account of model behaviour when that detail is necessary for a person to understand or challenge an outcome. The reviewed sources still stop short of requiring one named explanation technique, which is why the practical compliance problem remains selecting enough model-level and process-level evidence to make contestability real for the affected audience.

The evidence also points to audience-specific explanation as the practical operating model. A customer-facing explanation under GDPR is not the same artifact as a validator's challenge package under SR 11-7 or a technical dossier under the EU AI Act, even when they concern the same system. For regulated institutions, the most supportable operating model is layered explanation: one set of artifacts for affected individuals, another for supervisors and auditors, and another for internal engineering and model-risk teams.

Agentic systems remain the sharpest open edge because explanation scope now includes orchestration and tool-use provenance, not just model output rationale. Current mechanistic-interpretability work is valuable evidence that internal reasoning can sometimes be inspected, but it is not yet cheap, complete, or standardised enough to serve as the primary control for regulated deployment.

Risks, Gaps, and Uncertainties

Publicly accessible evidence for ISO/IEC 42001 is summary-level rather than clause-level, so this item does not make detailed claims about exact internal control wording.
The healthcare portion is narrower than the financial-services portion because the strongest accessible evidence in this session is the EU AI Act and official FDA lifecycle guidance, not a broad set of healthcare-sector supervisory statements.
The research question asks who "leads" XAI, but public evidence supports a distributed leadership answer better than a ranked list of top institutions.
Mechanistic interpretability is advancing quickly, so the current judgment that it is not yet a deployable compliance control could change materially within one or two research cycles.

Open Questions

Which explanation artifact set is sufficient for third-line audit of a multi-agent production workflow that spans multiple models and external tools?
Do any regulators move from technology-neutral explainability obligations toward named technical control expectations for agentic systems?
Which healthcare regulators outside the European Union and the United States publish the clearest operational expectations for explainability in AI-enabled clinical workflows?