Empirical Risk Minimisation's Causal Blindness

Empirical Risk Minimisation's Causal Blindness: Why In-Distribution Accuracy Guarantees Break Under Environment Change

2026-05-18 · ai-architecture benchmarks-eval causal-robustness epistemic-foundations · medium · source → · wiki →

key claims

ERM's formal PAC guarantee is distribution-conditional, because it bounds error only for hypotheses trained and evaluated on independent and identically distributed draws from the same underlying distribution rather than across environment changesBen-David (2014)Ben-David (2014)
That guarantee leaves causal structure unidentified, because a low-risk hypothesis may fit observational regularities without answering intervention or counterfactual questions about which feature actually generates the labelBen-David (2014)Pearl (2018)Meinshausen (2015)
A formal multi-environment counterexample shows that pooled ERM can rationally choose a spurious feature with lower average training error even when only the invariant feature retains low risk after the environment shiftsArjovsky et al. (2019)Scholkopf et al. (2021)
Spurious correlation is the mechanism of causal blindness under ERM, because minimizing empirical error rewards whichever cue predicts well on the observed sample whether that cue is structural or merely contextualArjovsky et al. (2019)Geirhos et al. (2020)
IRM is one formal correction proposed for ERM's blind spot because it searches for representations whose optimal classifier is invariant across training environments, which ties the learning objective more closely to stable causal structureArjovsky et al. (2019)Scholkopf et al. (2021)Meinshausen (2015)
Shortcut-learning evidence shows that benchmark-strong systems often solve tasks through background, context, or collection artifacts, so observed accuracy can coexist with a failure to learn the intended object-level ruleGeirhos et al. (2020)Geirhos et al. (2020)
Gradient-descent simplicity bias plausibly makes causally blind ERM solutions more likely in practice because optimisation can lock onto simple spurious features before it has to represent more complex invariant featuresYang et al. (2024)
This item therefore sharpens Research Question 1.3's instrumentalism critique by showing that prediction-first success is not merely philosophically incomplete but mathematically silent about whether the learned rule will travel beyond the observed regimeResearch (2026)Ben-David (2014)Arjovsky et al. (2019)

Research Question

How does the framework of Empirical Risk Minimisation (ERM) mathematically guarantee predictive accuracy within a known data distribution while remaining blind to the stable cause-and-effect relations needed to keep working after the data-generating environment changes?

Findings

Executive Summary

Empirical Risk Minimisation (ERM), the rule that chooses a model by minimising sample error, guarantees low error only for fresh examples drawn from the same distribution as the training sample under Probably Approximately Correct (PAC) learning, the framework that studies how sample performance transfers to new draws from that same distribution.

This is why ERM can be mathematically correct and still causally blind: the guarantee controls in-distribution risk, while causal robustness depends on whether the predictor tracks an invariant mechanism rather than a contingent correlation.

A formal counterexample shows that pooled ERM can prefer a spurious feature with lower training error even when only the noisier invariant feature survives environment shift.

Invariant Risk Minimisation (IRM), a multi-environment objective that requires the same optimal classifier across training environments, is one formal correction proposed in this literature, and gradient-descent simplicity bias helps explain why shortcut ERM solutions are often found first in practice.

Key Findings

ERM's formal PAC guarantee is distribution-conditional, because it bounds error only for hypotheses trained and evaluated on independent and identically distributed draws from the same underlying distribution rather than across environment changes.
That guarantee leaves causal structure unidentified, because a low-risk hypothesis may fit observational regularities without answering intervention or counterfactual questions about which feature actually generates the label.
A formal multi-environment counterexample shows that pooled ERM can rationally choose a spurious feature with lower average training error even when only the invariant feature retains low risk after the environment shifts.
Spurious correlation is the mechanism of causal blindness under ERM, because minimizing empirical error rewards whichever cue predicts well on the observed sample whether that cue is structural or merely contextual.
IRM is one formal correction proposed for ERM's blind spot because it searches for representations whose optimal classifier is invariant across training environments, which ties the learning objective more closely to stable causal structure.
Shortcut-learning evidence shows that benchmark-strong systems often solve tasks through background, context, or collection artifacts, so observed accuracy can coexist with a failure to learn the intended object-level rule.
Gradient-descent simplicity bias plausibly makes causally blind ERM solutions more likely in practice because optimisation can lock onto simple spurious features before it has to represent more complex invariant features.
This item therefore sharpens Research Question 1.3's instrumentalism critique by showing that prediction-first success is not merely philosophically incomplete but mathematically silent about whether the learned rule will travel beyond the observed regime.

Assumptions

[assumption] The constructed binary-feature counterexample is representative of the broader ERM failure class because the IRM paper's multi-environment examples and causal-invariance literature both license reasoning with stable versus unstable predictors even when the exact toy variables are chosen for clarity. [source: Arjovsky et al. (2019) Invariant Risk Minimization arxiv.org/abs/1501.01332]
[assumption] The arXiv and OpenReview versions consulted are materially faithful to the corresponding published arguments for the purposes of this item, because the seeded journal or DOI landing pages point to the same works and expose matching abstracts or metadata. [source: Geirhos et al. (2020) Shortcut Learning in Deep Neural Networks, Digital Object Identifier (DOI) record Shalev-Shwartz and Ben-David (2014) Understanding Machine Learning: From Theory to Algorithms, Digital Object Identifier (DOI) record Geirhos et al. (2020) Shortcut Learning in Deep Neural Networks arxiv.org/abs/1907.02893]

Analysis

ERM is not wrong on its own terms. It solves the problem it was asked to solve, namely selecting a low-risk hypothesis for a fixed sampling regime.

The difficulty is that mechanism and stability are external to that problem statement. Once deployment requires transfer across environments, the missing variable is no longer sample size alone but whether the predictor depends on invariant structure.

A plausible rival explanation is that OOD failures come mainly from poor data quality, poor regularisation, or weak evaluation, not from ERM itself. That rival explains part of the observed pathology, but it does not remove the core limitation because even perfect observational fit still leaves the causal identity of the predictive feature underdetermined.

Another rival explanation is that better optimisation or more data augmentation is enough. Those remedies can help, but the shortcut-learning and simplicity-bias evidence suggests they modify which correlations are easiest to use rather than proving that the selected rule is invariant by design.

Risks, Gaps, and Uncertainties

The most formal claim in this item is the scope of the PAC guarantee; the weakest part is the optimisation-bias explanation, which relies mainly on one recent primary source rather than on a mature multi-paper consensus.
The constructed counterexample is analytically clear but still illustrative rather than exhaustive, so it proves the possibility of causal blindness rather than its frequency across all domains.
IRM is included here as a formal correction, but this item does not evaluate when IRM itself fails, because comparative benchmark performance is outside scope.

Open Questions

Under what empirical conditions can a practitioner tell that a strong ERM model has discovered an invariant feature rather than a merely resilient shortcut?
Which environment-partitioning strategies give IRM enough heterogeneity to identify useful invariants in real deployment settings?
How should one compare causal robustness against the economic cost of collecting environment labels, interventions, or richer mechanistic priors?