Safety-alignment refusals cluster at early response tokens, enabling cheap selective intervention

Auto-published from arXiv:2609.36956 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [possible-zero-day, no-prompt-excerpt]) ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

alignment

Model

Gemini-3.5-Flash, GLM-5, Qwen3-32B, Kimi-K2.5

Surface

API (text-only, assistant-prefix continuation; Google API and OpenRouter)

Setup

The paper's empirical analysis measures KL divergence between the base model's next-token distribution and the distribution along successful jailbreak trajectories at every generation step. A Llama-3.1-8B-Instruct prefix-risk classifier (trained on the LLM-LAT dataset with Llama Guard labels) is used to identify high-risk positions. The risk-gated residual control component skips distribution reconstruction at positions the classifier marks as safe, concentrating the ~50-sample reconstruction budget only at positions where refusal is likely. No verbatim prompt is provided in the paper.

Reproduction anchor

code: https://github.com/JessonWong/controlled-decoding (cited in HTML body); Algorithm 1 in paper body contains the full inference pseudocode; prefix-risk classifier backbone: Llama-3.1-8B-Instruct, trained on LLM-LAT dataset

Observed behavior

Large distributional changes between base and jailbroken trajectories are concentrated at a small subset of token positions — primarily opening tokens of the response. The risk-gated mechanism exploits this to skip reconstruction at the majority of positions, substantially reducing query cost while maintaining jailbreak effectiveness comparable to always-on reconstruction.

Expected behavior

Safety alignment should produce distributed refusal signals throughout generation, not depend on a narrow prefix window; a robust alignment mechanism would prevent a small number of early interventions from unlocking harmful continuations.

Reproducibility

medium

Threat model

The finding that alignment is structurally concentrated at early tokens reveals a systemic weakness exploitable by any attacker who can influence the first few tokens of an assistant response (e.g., via assistant-prefix APIs, prefix injection, or system-prompt manipulation). This reduces the economic cost of black-box attacks — fewer samples needed — making large-scale automated harmful-content extraction viable.

Novelty

First empirical characterization of where in the token sequence safety-alignment 'fires', showing that alignment is not uniformly distributed across generation and can be defeated by selectively targeting a small early window, with implications for alignment robustness evaluation.

Source

• arXiv: 2609.36956
• PDF: https://arxiv.org/pdf/2609.36956
• Categories: cs.CR, cs.AI
• Authors: Jesson Wang, Shawn Li, Wei Yang, Franck Dernoncourt, Ryan A. Rossi, Charith Peris, Yue Zhao

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Novel attack framework for jailbreaking black-box LLMs via text-only interfaces; tested against 4 deployed endpoints with demonstrated bypasses of safety alignment. No vendor disclosure mentioned.

─────────────────────────────────────────────────────────────────────