Pre-Training Origins of Hallucination-Associated Neurons — Implications for LLM…

Research Question

Given that Hallucination-Associated Neurons (H-Neurons) emerge during pre-training rather than instruction tuning or RLHF, what does this reveal about how hallucination-prone behaviour is encoded during the pre-training phase, and what concrete changes to pre-training data, objectives, or architecture could reduce H-Neuron formation?

Findings

Executive Summary

H-Neurons originate during pre-training because the next-token prediction objective provides an unrestricted positive reward for compliant, agreeable continuations whenever training corpora contain sycophantic or confidently-stated-false text — which web-scale corpora do at sufficient concentration to consolidate dedicated over-compliance circuits. Post-training alignment (SFT, RLHF) cannot eliminate these circuits: Gao et al. confirm that H-Neurons exhibit parameter inertia, changing minimally during alignment, and the RLHF literature confirms that reward signals operate at the response-distribution level rather than rewriting individual pre-trained circuits. Scaling reduces H-Neuron ratios and per-neuron perturbation leverage, but distributes rather than eliminates the compliance feature, explaining why even frontier models hallucinate despite aggressive alignment. Three pre-training intervention families have candidacy: data quality filtering (strongest evidence, operational precedent from Phi-1 and RefinedWeb), auxiliary uncertainty-expression objectives (most targeted, unvalidated at scale), and architectural superposition reduction (most speculative, lowest supporting evidence). No intervention has been validated by a direct pre-training ablation for H-Neuron formation specifically; this remains the primary gap in the literature.

Key Findings

1. H-Neurons are present and predictive in base pre-trained models before any instruction tuning or RLHF, confirmed by backward-transferability experiments in Gao et al.: classifiers built from aligned-model H-Neurons achieve AUROC scores significantly above random baseline on all six base models and three evaluation domains (TriviaQA, BioASQ, NQ-Open). [confidence: high]

2. H-Neurons exhibit parameter inertia during SFT: they rank among the model parameters that change least during the transition from base to instruction-tuned model, with average cosine-similarity rank of ≈0.97 in Mistral-Small (P < 0.001), confirming that standard instruction tuning does not restructure the hallucination circuits. [confidence: high]

3. The next-token prediction objective is the proximate causal mechanism for H-Neuron formation, because it rewards fluent agreeable continuations without a countervailing factual accuracy penalty; on training corpora containing sycophantic and confidently-stated-false text, this signal consolidates dedicated over-compliance circuits in FFN neurons. [confidence: high for causal attribution to NTP objective; medium for the specific role of sycophantic corpus content, which is inferred rather than ablated]

4. RLHF suppresses H-Neuron expression at the output level but does not rewrite the underlying circuits: RLHF gradient updates are diffuse and of limited magnitude relative to the billions of NTP steps that consolidated H-Neurons, and the alignment tax constrains how aggressively reward signals can push against pre-trained features without capability degradation. [confidence: high]

5. Larger models have lower H-Neuron ratios (0.01‰ in 24–70B models vs. 0.35‰ in 7B models) and lower per-neuron compliance perturbation slopes (average 2.40 vs. 3.03), indicating that the over-compliance feature is more diffusely encoded at scale — consistent with superposition theory — rather than eliminated. [confidence: medium — based on six model checkpoints across three families]

6. Scaling training tokens and model size equally (Chinchilla-optimal) increases pre-training exposure to compliance-inducing corpus patterns proportionally with model capacity, so scale alone does not resolve the H-Neuron formation problem even as it reduces per-neuron perturbation leverage. [confidence: medium — inference from Chinchilla scaling law applied to compliance-signal volume]

7. Data quality filtering — domain-level source weighting and removal of low-reliability sycophantic content — is the pre-training intervention with the strongest supporting evidence: Phi-1 demonstrates that 7B tokens of textbook-quality data outperforms 300B tokens of unfiltered web data on coding benchmarks, and RefinedWeb shows that aggressive CommonCrawl filtering matches curated-corpus performance at 5 trillion tokens. [confidence: medium — evidence is from coding/general capability, not H-Neuron formation specifically]

8. An auxiliary uncertainty-expression pre-training objective — rewarding "I don't know" responses as strongly as correct answers on a fraction of training examples — would directly counteract the NTP compliance reward and is the most mechanistically targeted intervention, but has not been validated at frontier training scale and carries risk of degrading task-completion capability if poorly calibrated. [confidence: low — inference from first principles]

9. Architectural reduction of superposition pressure (widening FFN intermediate layers, auxiliary sparsity losses) could reduce H-Neuron consolidation by allowing compliance features to be more orthogonally encoded, but this prediction is derived from toy-model superposition theory and has no direct validation in pre-training experiments with large transformer models. [confidence: low]

Assumptions

• Assumption: Web-scale pre-training corpora contain sycophantic and confidently-stated-false text at sufficient concentration and consistency to consolidate dedicated over-compliance circuits. Justification: The Constitutional AI paper (Anthropic 2022) motivates its own RLHF approach partly by noting that pre-training data embeds compliance-encouraging patterns. The Phi-1 paper's result that textbook-quality data produces dramatically different model behaviour from unfiltered web data is consistent with data composition having large effects on behavioural dispositions. No direct ablation study removing sycophantic content and measuring H-Neuron formation has been published.

• Assumption: The parameter-inertia finding for SFT extends to RLHF. Justification: The paper tests SFT models only. RLHF involves more extensive parameter updates, but the structural argument — that NTP-consolidated circuits are more resistant to diffuse gradient updates than to the original concentrated training signal — applies to both. The jailbreak-recovery literature (where RLHF-aligned models revert to pre-training behaviour under adversarial prompts) supports this assumption empirically.

• Assumption: The mechanistic insights from Phi-1 (coding quality data) generalise to factual-accuracy and compliance training. Justification: The Phi-1 result is in the coding domain; the generalisation to natural language factual accuracy is an inference. The same filtering logic applies, but the specific relationship between textbook-quality text and over-compliance reduction is not directly validated.

Analysis

The evidence base for the pre-training origin claim is strong: two complementary experiments (backward transferability and parameter evolution) in Gao et al. both point in the same direction, and neither is easily dismissed by alternative hypotheses. The mechanistic attribution — NTP objective on compliance-laden corpora produces H-Neurons — is well-supported by the theoretical literature (Kalai et al.) and is consistent with the Phi-1 and RefinedWeb data quality findings, but lacks a direct ablation experiment. This gap matters for intervention design: if the mechanism is confirmed by ablation, data filtering becomes a high-priority action; without it, the intervention-evidence chain rests on inference.

The RLHF limits argument is the most robust of the cross-source claims. It is supported by: (1) Gao et al.'s parameter inertia finding, (2) the Constitutional AI paper's own framing (it explicitly treats RLHF as layered on top of pre-training, not a replacement), (3) the alignment tax literature, and (4) the jailbreak-recovery empirics. These four independent sources converging on the same conclusion raises confidence above what any single source would provide.

The scaling picture requires careful handling. The claim that "larger models hallucinate less" is true for absolute hallucination rates but does not imply the structural cause is reduced. The H-Neuron ratio data and compliance-slope data together support a model in which the compliance feature is more diffusely encoded at scale — consistent with superposition theory — rather than eliminated. This interpretation is the most coherent reading of the available evidence, though the small number of data points (six model checkpoints) warrants medium rather than high confidence.

Intervention feasibility varies substantially by category. Domain-level quality weighting is already standard practice (Phi-1, RefinedWeb, FineWeb) and has the strongest supporting evidence, though that evidence is not directly about H-Neuron formation. Auxiliary uncertainty-expression objectives are mechanistically optimal but technically undeveloped at frontier scale. Architectural changes are the most speculative.

Risks, Gaps, and Uncertainties

• Primary gap: No direct pre-training ablation experiment tests whether removing sycophantic content or adding uncertainty-expression objectives reduces H-Neuron formation. The causal inference from NTP objective to H-Neuron formation is theoretically motivated and empirically consistent, but not directly confirmed by a pre-training intervention experiment.

• Scope gap: Gao et al. test SFT (instruction-tuned) models, not RLHF-trained models. The assumption that RLHF is equally ineffective at modifying H-Neurons is well-motivated but not directly tested.

• Scaling evidence gap: The scaling pattern (lower ratio, lower perturbation slope in larger models) is based on six model checkpoints across three families. It is consistent with superposition theory but is not a validated scaling law for H-Neuron density.

• Inaccessible source: Kalai et al. (2025) "Why language models hallucinate" is cited in Gao et al. but the full text was not directly accessed for this research item. The claim attributed to this paper (hallucination is an inevitable learning-theoretic consequence of NTP) is taken from Gao et al.'s characterisation of it.

• Uncertainty about data filtering scope: Even if sycophantic text removal reduces H-Neuron formation, the practical difficulty of identifying and filtering sycophantic patterns at the scale of 5 trillion tokens without introducing new biases is unresolved.

Open Questions

• Is there a variant of the next-token prediction objective that treats factual uncertainty as a first-class training signal — e.g., by reserving a fraction of training examples for which uncertainty expression is the correct response? Priority: medium (directly actionable for model developers; no existing validation at frontier scale).
• Can "H-Neuron density" become a reportable model evaluation metric, analogous to perplexity or benchmark accuracy, to incentivise pre-training investment in compliance-reduction? Priority: medium (requires standardised measurement protocol; feeds into regulatory discourse).
• If sycophantic web text is the primary corpus signal driving H-Neuron formation, does synthetic data generation in the style of Phi-1 ("textbooks") measurably reduce H-Neuron formation compared to filtered web data of equivalent volume? Priority: high (directly tests the causal hypothesis; could be run without frontier-scale compute using Pythia-style controlled pre-training experiments).

sources

• [x] Gao et al. (2025) — "Hallucination-Associated Neurons in LLMs" (primary source for origins finding)
• [x] Hoffmann et al. (2022) — "Training Compute-Optimal Large Language Models (Chinchilla)" — scaling laws and data quality
• [x] Gunasekar et al. (2023) — "Textbooks Are All You Need" (Phi-1) — data quality over data scale
• [x] Penedo et al. (2023) — "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Alone" — pre-training data curation
• [x] Anthropic (2022) — "Constitutional AI: Harmlessness from AI Feedback" — RLHF limits for alignment
• [x] Elhage et al. (2022) — "Toy Models of Superposition" — how pre-training encodes features in neurons
• [x] Biderman et al. (2023) — "Pythia: A Suite for Analysing Large Language Models Across Training and Scaling" — training dynamics
• [ ] Research/backlog/2026-03-05-h-neurons-in-llms.md — prerequisite: must be completed before starting this item