Safety alignment's refusal mechanisms fail to activate on code-domain prompts — confirmed by activation steering

Auto-published from arXiv:2609.39902 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [possible-zero-day, no-prompt-excerpt])

Category

alignment

Model

GPT-4o (1120), GPT-4.1 (0414), GPT-5-chat (1003), Claude-3.7-Sonnet (20250219), Claude-Sonnet-4 (20250514), Gemini-2.5-Pro, Llama-3.1-70B-Instruct, Llama-3.3-70B-Instruct

Surface

API (black-box); latent-space analysis performed on accessible models

Setup

The authors project the internal representations of CodeMimicry jailbreak prompts onto refusal-related latent directions and apply activation steering (steering representations toward or away from the 'jailbreak direction'). They compare successful jailbreaks, failed jailbreaks, benign code prompts, and natural-language harmful prompts using t-SNE visualization. The paper does not provide the exact steering vectors or layer indices in accessible text; Appendix details are referenced.

Reproduction anchor

code: https://github.com/lzzzr123/CodeMimicry — latent space analysis described in Section 5 of the paper; activation steering methodology referenced in the body

Observed behavior

Successful CodeMimicry jailbreak prompts cluster with benign code in latent space and show significantly lower projection onto refusal directions than failed jailbreaks or natural-language harmful queries. Activation steering away from the jailbreak direction restores refusals, confirming that the safety mechanism is present but systematically not activated by code-domain inputs. Failed jailbreaks form isolated clusters distinct from both benign code and successful jailbreaks.

Expected behavior

Refusal-related activation directions should fire on harmful content regardless of whether that content is expressed in natural language or structured code; the safety alignment should generalize across surface representations.

Reproducibility

medium

Threat model

Safety teams and model providers that assume RLHF/SFT safety alignment transfers uniformly across input modalities are systematically wrong for code inputs; this structural gap will persist in any model trained primarily on NL safety examples, enabling any attacker who discovers this pattern to reliably bypass alignment on production systems.

Novelty

First mechanistic account of why code-based jailbreaks succeed: they navigate the latent space to remain indistinguishable from benign code along refusal-related axes, exposing 'safety generalization lag' as a structural training deficiency rather than an implementation bug that can be patched by simple filters.

Source

• arXiv: 2609.39902
• PDF: https://arxiv.org/pdf/2609.39902
• Categories: cs.CR, cs.AI
• Authors: Zhen Liang, Hai Huang, Wentao Chen

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Identifies a novel jailbreak vulnerability (safety generalization lag in code domains) with an automated exploitation method (CodeMimicry) achieving 96% success on 8 commercial LLMs. Includes mechanistic analysis of how the vulnerability bypasses safety mechanisms. No disclosure coordination mentioned.

─────────────────────────────────────────────────────────────────────