LLM Response Style and Confidence Signalling

LLM Response Style and Confidence Signalling: How AI Fluency Degrades User Calibration on Uncertainty in Ambiguous Policy Compliance Contexts

2026-05-17 · governance-policy security-risk benchmarks-eval organisational-design · medium · source → · wiki →

key claims

Natural-language uncertainty expressions can materially improve whether user reliance matches answer reliability, because participants exposed to first-person LLM hedging relied on the system less blindly and answered medical questions more accurately than participants shown more assertive wordingKojima et al. (2024)
Explanations and polished answer structures do not automatically create better human-AI decisions, and multiple studies show they can increase acceptance of recommendations even when the underlying recommendation quality does not improveBansal et al. (2021)Schubert et al. (2023)
Information that is easier to process because it is repeated or smoothly presented can raise subjective confidence independently of factual truth, which means a polished policy explanation can feel more reliable than it is before any substantive validation occursStump et al. (2024)
Current LLMs are not dependable narrators of their own certainty, because recent benchmark studies find persistent overconfidence and large gaps between stated confidence and actual correctness even when prompt engineering and consistency checks improve performance somewhatXiong et al. (2024)Shukla et al. (2024)
On legal-style questions that resemble policy interpretation more closely than general trivia does, state-of-the-art LLMs still produce frequent hallucinations, fail to correct false premises reliably, and often need extra probing before their uncertainty becomes visibleDahl et al. (2024)Agrawal et al. (2024)
Ambiguity handling skill does not eliminate interpretation risk, because Kamath et al. report over 90% accuracy in some ambiguity datasets while other studies still show weak matching between stated confidence and actual correctness, plus frequent legal-task hallucinations in real user-facing settingsKamath et al. (2024)Dahl et al. (2024)Xiong et al. (2024)
For policy and compliance assistants, the main governance control should be preserving the user's motivation to verify or escalate, not merely improving prose quality, because wording, aggregation, queue pressure, and the risk of under-reliance all affect whether human review remains meaningfulNational (2023)Schubert et al. (2023)Github (n.d.)Github (n.d.)Dietvorst et al. (2015)

Research Question

How do Large Language Model (LLM) response style and self-reported confidence change how accurately users judge uncertainty and downstream risk when interpreting ambiguous policy and compliance requirements?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

Confident, fluent LLM policy answers are likely to make users underweight ambiguity and overestimate correctness unless the system explicitly signals uncertainty and keeps evidence inspection visible.

The strongest direct evidence is at the user-interface layer: natural-language uncertainty expressions reduce overreliance, while explanations and polished presentation can increase acceptance of advice without improving correctness.

Current LLM self-reported confidence is too weakly matched to actual correctness to substitute for external verification on its own, because recent evaluations show overconfidence and weak error awareness on difficult or legal-style tasks even when some ambiguity-handling capability exists.

For enterprise policy and compliance assistants, a safer default control pattern is uncertainty-forward wording that is matched to evidence and task stakes, direct quotation of governing text, and escalation of ambiguous or high-impact cases rather than a single authoritative answer path.

Key Findings

Natural-language uncertainty expressions can materially improve whether user reliance matches answer reliability, because participants exposed to first-person LLM hedging relied on the system less blindly and answered medical questions more accurately than participants shown more assertive wording.
Explanations and polished answer structures do not automatically create better human-AI decisions, and multiple studies show they can increase acceptance of recommendations even when the underlying recommendation quality does not improve.
Information that is easier to process because it is repeated or smoothly presented can raise subjective confidence independently of factual truth, which means a polished policy explanation can feel more reliable than it is before any substantive validation occurs.
Current LLMs are not dependable narrators of their own certainty, because recent benchmark studies find persistent overconfidence and large gaps between stated confidence and actual correctness even when prompt engineering and consistency checks improve performance somewhat.
On legal-style questions that resemble policy interpretation more closely than general trivia does, state-of-the-art LLMs still produce frequent hallucinations, fail to correct false premises reliably, and often need extra probing before their uncertainty becomes visible.
Ambiguity handling skill does not eliminate interpretation risk, because Kamath et al. report over 90% accuracy in some ambiguity datasets while other studies still show weak matching between stated confidence and actual correctness, plus frequent legal-task hallucinations in real user-facing settings.
For policy and compliance assistants, the main governance control should be preserving the user's motivation to verify or escalate, not merely improving prose quality, because wording, aggregation, queue pressure, and the risk of under-reliance all affect whether human review remains meaningful.

Assumptions

Evidence from legal, medical, and personnel-selection decision support transfers well enough to enterprise policy interpretation because the relevant mechanism is human reliance on uncertain recommendations rather than the substantive domain alone.
Internal policy assistants will often operate in organisations where escalation is slower than answering, so reliance and bypass incentives are practical governance concerns rather than only laboratory effects.

Analysis

The evidence is strongest on two points: users change their verification behavior when wording or interface cues change, and current LLMs remain weak at matching stated confidence to actual correctness.

The main alternative explanation is that better explanations or better model capability could remove the need for explicit uncertainty or escalation controls. The retrieved evidence does not support that stronger claim, because explanations increased acceptance without improving complementary performance, and ambiguity competence did not come with proven confidence-quality matching or error self-awareness.

I therefore weighted evidence about how wording changes user reliance more heavily than raw ambiguity-performance evidence when answering the research question. This does not imply that stronger uncertainty cues should be maximised without limit, because Dietvorst's algorithm-aversion study and later reliance-adjustment work both show that visible error or poorly tuned cues can also produce under-reliance. For governance design, the relevant failure is not only whether the model can sometimes get ambiguity right, but whether users can tell when they should stop trusting the first answer and seek authoritative review without being pushed into blanket disuse.

Risks, Gaps, and Uncertainties

The retrieved literature did not include a direct experiment on ambiguous internal corporate policies, so the conclusion relies on adjacent decision-support domains rather than a perfect task match.
LLM confidence-estimation methods are moving quickly, so absolute error rates may change faster than the broader human-factors pattern.
Policy interpretation risk will vary by task type, because some ambiguity classes are easier for models than open-ended legal or organisational interpretation tasks.
Organisational incentives may dominate interface improvements if escalation remains materially slower or more costly than accepting the assistant's answer.

Open Questions

Does numeric confidence plus direct quotation of the governing policy text outperform verbal hedging alone for enterprise users?
Which escalation trigger best preserves verification without making the assistant unusably slow: ambiguity detection, policy-domain risk tiering, or explicit user uncertainty declarations?
Can audit logs capture whether users inspected source text before acting on a policy answer strongly enough to support meaningful review?