Single-turn safety benchmarks miss a 19x divergence between GPT-4.1-mini and Claude Sonnet 4.5 under identical multi-turn adversarial pressure

Auto-published from arXiv:2606.02630 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria clear: models named precisely, attack system prompt reproduced in appendix against a named public dataset with a named attacker model, 19x Turn-4 divergence (78.8% vs 4.1%) is a sharp observed-vs-expected gap, medical-AI certification blind spot is a concrete real-world consequence, and controlled equivalence-at-Turn-1 design is genuinely novel. needsDisclosureReview is mitigated: paper is already public on arxiv and triage scored possibleZeroDay false. Only flag is per-item prompts are LLM-generated and not archived, reducing reproducibility to medium. ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

alignment

Model

GPT-4.1-mini

Surface

chat API

Setup

The same live adversarial attacker (GPT-4o-mini) and same 464-item PatientSafetyBench dataset are applied to both GPT-4.1-mini and Claude Sonnet 4.5 across four turns. At Turn 1, the two models are statistically indistinguishable (unsafe rates 34.8% vs 31.8%, p=0.43). No verbatim per-item prompts are published, but the attack system prompt is provided in the appendix.

Observed behavior

GPT-4.1-mini follows a "compliance creep" trajectory (34.8% → 77.7% → 77.5% → 78.8% unsafe), while Claude Sonnet 4.5 follows a "pattern recognition" trajectory (31.8% → 28.3% → 10.5% → 4.1% unsafe). By Turn 4, the gap is 19x (78.8% vs 4.1%), a difference entirely invisible to standard single-turn evaluation.

Expected behavior

Models certified as equivalent on single-turn safety benchmarks should show comparable multi-turn degradation, or safety certification pipelines should detect the divergence before deployment.

Reproducibility

medium

Threat model

Developers and compliance teams who use single-turn safety benchmarks to certify medical AI for production deployment may ship systems with catastrophic multi-turn degradation. An adversarial patient would need only four conversational turns to push an apparently-safe system to a 79% unsafe response rate.

Novelty

First controlled demonstration that two models statistically equivalent on single-turn safety can diverge by 19x under identical multi-turn adversarial pressure, exposing a systematic blind spot in the current standard for AI safety certification.

Source

• arXiv: 2606.02630
• PDF: https://arxiv.org/pdf/2606.02630
• Categories: cs.CR, cs.AI
• Authors: Anushka Sheoran, Yiduo Hao

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 3
• triage reason: MultiTurnPSB is a concrete benchmark demonstrating specific, quantified jailbreak failures (35%→80% unsafe responses via multi-turn attacks) on GPT-4.1-mini and Claude Sonnet 4.5. The paper characterizes failure signatures and identifies reproducible attack patterns, making it case-rich.

─────────────────────────────────────────────────────────────────────