RAG+evidence-verification pipeline increases over-refusal to 100% vs. 98.3% base rate in GPT-4o

Auto-published from arXiv:2610.01497 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.76, flags: [other])

Category

over-refusal

Model

GPT-4o (gpt-4o-2024-08-06)

Surface

API with RAG pipeline (CIViC records, FDA approvals; temperature=0, max_tokens=512)

Setup

GPT-4o was evaluated under two configurations on the OpenMTB-Audit benchmark: (a) Base — direct prompting with no retrieval; (b) RAG+EV — augmented with retrieved CIViC biomarker-therapy concordance records and evidence verification step before classification. Both received the same 60 ground-truth 'Partially Supported' NSCLC cases. The 'Unsafe Overconfidence' adversarial category is particularly relevant: these cases involve a molecularly matched therapy that is evidence-backed but requires oncologist review due to poor ECOG performance status (≥3) or extensive prior treatment. Full prompt templates are in the linked repository.

Reproduction anchor

code: https://github.com/Negiiinx/OpenMTB-Audit — benchmark cases, RAG pipeline code, and prompt templates; paper Table 2 reports per-configuration over-refusal rates

Observed behavior

GPT-4o Base had 98.3% over-refusal on Partially Supported cases (accuracy 67.6%, safety score 76.4). GPT-4o RAG+EV had the same 98.3% over-refusal but lower overall accuracy (55.4%) and higher safety score (90.9) — meaning retrieval increased apparent safety while reducing actual correctness. The Prompt-Checklist variant reached 100% over-refusal with 74.0% accuracy and 90.1 safety score. Adding evidence retrieval did not help models distinguish 'Partially Supported' from 'Unsupported'.

Expected behavior

A RAG-augmented pipeline with access to evidence records should be better-positioned to recognize that a therapy with biomarker support but clinical caveats warrants 'Partially Supported' rather than 'Unsupported', since it has explicit access to evidence strength and patient context.

Reproducibility

high

Threat model

Developers augmenting clinical LLMs with retrieval-augmented generation may assume richer evidence access improves safety calibration; this paper shows the opposite — RAG pipelines in production oncology tools can increase over-conservatism, leading to systematically misleading safety audits that obscure the failure mode from downstream validators.

Novelty

Counterintuitive finding that retrieval-augmented evidence verification amplifies over-refusal rather than correcting it, exposing a failure mode specific to multi-class clinical safety classification where RAG surfaces contraindications but lacks the reasoning structure to apply them conditionally.

Source

• arXiv: 2610.01497
• PDF: https://arxiv.org/pdf/2610.01497
• Categories: cs.AI, cs.LG
• Authors: Negin Ashrafi, Jia Luo, Stacey M. Frumm, Roxana Daneshjou

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 2
• triage reason: Benchmark of 500 clinical cases exposing reproducible over-refusal failure: all 8 LLM configurations collapse fine-grained labels (Partially Supported) in 83.3–100% of cases rather than performing calibrated clinical reasoning.

─────────────────────────────────────────────────────────────────────