SmoothLLM defense amplifies SN-Guided jailbreak ASR on Llama-3-8B from 86% to 95%

Auto-published from arXiv:2608.07430 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, flags: [possible-zero-day])

Category

alignment

Model

Llama-3-8B-Instruct

Surface

API with SmoothLLM defense wrapper

Setup

SmoothLLM — a randomized smoothing defense that adds character-level perturbations to inputs and majority-votes over multiple noisy copies — is applied as a post-hoc defense against SN-Guided Diffusion jailbreak prompts. The jailbreak prompts are generated offline using the persona-masked diffusion template against the Llama-3-8B-Instruct target. The SmoothLLM wrapper is then interposed between the attacker's prompt and the model at inference time.

Reproduction anchor

code: https://github.com/ellyoana/sn-guided-diffusion; defense evaluation in Table 4 of the paper; 100 prompts from JailBreakV-28K used

Observed behavior

Rather than reducing ASR, SmoothLLM increased ASR on Llama-3-8B-Instruct from 86.0% to 95.0%. The perturbations introduced by SmoothLLM appear to destroy the coherence of safety-relevant token patterns while preserving the harmful semantic content, net-worsening alignment. Perplexity filtering had negligible impact (86.0% → 85.0%), and only layer-specific editing provided meaningful mitigation (reducing ASR to 69.0%).

Expected behavior

A defense wrapper like SmoothLLM should reduce or hold steady the fraction of harmful completions; it should not amplify jailbreak success.

Reproducibility

medium

Threat model

Organizations deploying SmoothLLM (or similar randomized-smoothing wrappers) as a drop-in safety layer for production LLM APIs may inadvertently increase their models' susceptibility to diffusion-generated jailbreaks. A sophisticated attacker aware of this interaction could deliberately target SmoothLLM-hardened endpoints, expecting amplified success rates.

Novelty

Concrete demonstration that a widely-used alignment defense (SmoothLLM) is counter-productive against diffusion-based jailbreaks, exposing a defense-failure mode specific to the masked-token structure of SN-Guided prompts.

Source

• arXiv: 2608.07430
• PDF: https://arxiv.org/pdf/2608.07430
• Categories: cs.LG, cs.AI
• Authors: Elena Dumitrescu, Gert Lek, Lydia Y. Chen, J'er'emie Decouchant

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Discloses mechanistic jailbreak vulnerabilities against frontier LLMs including Gemini-2.5-Flash-Lite through novel safety neuron attacks with reproducible setups and concrete ASR metrics.

─────────────────────────────────────────────────────────────────────