GradSafe calibrated on synthetic benign data flags over 90% of WildChat benign conversations as unsafe

Auto-published from arXiv:2609.36849 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt, possible-zero-day])

Category

over-refusal

Model

unknown

Surface

API safety filter (GradSafe + Context Window Scanner)

Setup

GradSafe detection threshold is calibrated using synthetic benign conversations (standard prior-work practice), then evaluated against WildChat — a large dataset of real-world user conversations. No specific prompt excerpt is provided in the abstract.

Reproduction anchor

benchmark: WildChat (public HuggingFace dataset); GradSafe detector (published). No artifact URL for this specific experiment found in fetched content.

Observed behavior

ROC-AUC drops from 0.98 (synthetic benign baseline) to 0.76 (WildChat benign baseline). A threshold calibrated on synthetic data flags more than 90% of real WildChat benign conversations as unsafe.

Expected behavior

Low false-positive rate on realistic benign traffic; AUC should not degrade catastrophically under distribution shift.

Reproducibility

medium

Threat model

Any production system deploying gradient-based jailbreak detection calibrated on synthetic benign data will block legitimate users at catastrophic rates under organic traffic, creating a practical denial-of-service for real users and making the defense undeployable without domain-matched calibration.

Novelty

Quantifies with concrete numbers (AUC 0.98 to 0.76, >90% FPR) the distributional gap between synthetic and realistic benign traffic for gradient-based safety detectors, establishing a reproducibility failure mode for the entire detector class.

Source

• arXiv: 2609.36849
• PDF: https://arxiv.org/pdf/2609.36849
• Categories: cs.CR, cs.AI, cs.IR, cs.LG
• Authors: Omar Sheta, Rinku Deuja, Hadi Masoudi, Minghong Fang

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Paper systematically demonstrates failures of gradient-based jailbreak detection (GradSafe) in multi-turn dialogue—a novel attack vector. Provides reproducible multi-turn evasion cases, false-positive miscalibration on realistic benign data (90% flagging rate), and model-dependent detector failures.

─────────────────────────────────────────────────────────────────────