Users' trust in GPT-5-mini tutor does not drop as accuracy falls from 74% to 29%

Auto-published from arXiv:2610.31579 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])

Category

alignment

Model

GPT-5-mini

Surface

Custom chat UI (React/Node/Express/SQLite) wrapping OpenAI Chat Completions API

Setup

Same HARICHAN study design. After each learning session, participants rated the assistant on perceived accuracy and willingness to rely on it again (both on 1–7 Likert scales). The assistants operated at 0%, 25%, 50%, 75%, and 100% falsification, corresponding to objective quiz accuracy dropping from 7.4 to 2.9 out of 10. The assistant never hedged, never conceded under challenge (0/16 challenges resulted in concession), and always responded to accuracy questions with a fixed deflection.

Reproduction anchor

prompt: Appendix A (full system prompt) and Appendix B (fact pairs). No public code or data repository URL found in the paper.

Observed behavior

Post-task perceived accuracy ratings declined only from 5.1 to 4.5 across the full 0%–100% falsification range (slope b=−0.12, p=.14, not significant). Willingness to rely on the assistant again was flat at 4.6–4.9 (p=.82). Within-participant trust sensitivity had a median slope of 0.00, with 11 of 32 participants showing flat or inverted trust–accuracy relationships. Metacognitive discrimination (AUROC₂) collapsed from 0.70 to near-chance (0.52–0.53) at the highest falsification levels; expected calibration error rose from 0.13 to 0.52.

Expected behavior

Trust and willingness-to-rely ratings should decrease meaningfully as the assistant's objective accuracy degrades. Users' confidence calibration (AUROC for own answers) should remain above chance regardless of assistant quality.

Reproducibility

medium

Threat model

Deployed AI assistants that present all outputs with uniform high confidence prevent users from developing accurate reliability intuitions, even across extended interactions. This is especially dangerous in high-stakes domains (medical, legal, educational) where users are likely to rely on the assistant precisely when their own domain knowledge is insufficient to catch errors.

Novelty

Demonstrates empirically that trust ratings and self-confidence calibration are immune to objective accuracy degradation when the AI maintains uniform confident tone — collapsing the assumption that user trust is a useful signal for model reliability monitoring.

Source

• arXiv: 2610.31579
• PDF: https://arxiv.org/pdf/2610.31579
• Categories: cs.AI, cs.HC
• Authors: Deepthika S, Pawni Yadav, B. Sankar

Triage notes (auto)

• paperType: case-study
• estimatedCaseCount: 1
• triage reason: Demonstrates reproducible failure with controlled experimental setup: confident hallucinations from GPT-5-mini cause users to adopt false statements ~65% of the time and retain them with high confidence (8.4/10), with measurable collapse of confidence-as-correctness signal (AUROC 0.70→chance).

─────────────────────────────────────────────────────────────────────