Qwen2.5 and Gemma-2-9B merged models show 60–76% adaptive ASR while Llama-3.1-8B stays at ~24% under identical attack

Auto-published from arXiv:2608.08542 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.85, flags: [no-prompt-excerpt])

Category

alignment

Model

Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Gemma-2-9B (all skill-merged variants); contrast with Llama-3.1-8B-Instruct and Phi-4-mini

Surface

API / inference harness (SkillSafe-Bench cross-family evaluation)

Setup

Six open-weight base models across five families are merged with math and/or uncensored skill task vectors using four algorithms (task arithmetic, TIES, DARE-TIES, linear) at five coefficients (λ ∈ {0.2, 0.4, 0.6, 0.8, 1.0}). All resulting merges are evaluated with the same best-of-6 semantic template attack (Appendix S2) on a fixed 50-behavior HarmBench subset. The paper measures adaptive ASR gap = (adaptive ASR) − (static ASR) per base family. No gradient access or model internals required by the attacker.

Reproduction anchor

benchmark: HarmBench standard text behaviors (n=400 static, n=50 adaptive subset); prompt: Appendix S2 (six verbatim templates). Paper cites a code supplement mapping each table to result JSONs but no public repository URL was located.

Observed behavior

Under the semantic template attack, Qwen2.5-7B and Qwen2.5-14B merges reach 60–76% adaptive ASR across all merging recipes, with static-to-adaptive gaps of ~20 percentage points. Gemma-2-9B merges similarly fall in the 60%+ range. Llama-3.1-8B-Instruct merges stay at ~24% adaptive ASR (gap ~18pp but from a lower base), and Phi-4-mini remains comparably robust. The fragility pattern is base-conditional: it persists regardless of which merging algorithm or coefficient is used, indicating the vulnerability is a property of the base model's safety geometry, not the merge recipe.

Expected behavior

Safety evaluations for merged models should be predictive of adaptive attack robustness; a model that passes static screening should not be trivially broken by public-template attacks at 60–76% rates.

Reproducibility

medium

Threat model

An AI developer or enterprise adopts a community-released skill-merged Qwen or Gemma model that passed standard safety benchmarks (e.g., HarmBench static, MT-Bench). A downstream attacker or red-teamer applies freely available role-play jailbreak templates and achieves harmful-content extraction at rates that would have been rejected pre-deployment — harm scales with the breadth of deployment and skill of the merged model (e.g., medical, legal, or chemistry domain specialists).

Novelty

Demonstrates that adaptive jailbreak fragility is a base-model property, not a merge-recipe artifact: certain model families (Qwen, Gemma) are structurally vulnerable across all tested merge configurations, while others (Llama, Phi-4) are structurally robust — providing the first cross-family characterization of this effect at scale.

Source

• arXiv: 2608.08542
• PDF: https://arxiv.org/pdf/2608.08542
• Categories: cs.LG
• Authors: Yu Ma, Hongli Shi, Jing Li, Xinran Xu, Weiwei Hou

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 2
• triage reason: SkillSafe-Bench is a controlled benchmark that systematically demonstrates concrete jailbreak failures of skill-merged LLMs (Qwen, Gemma at 60–76% rates) via semantic template attacks, despite static refusal appearing safe. This is a primary benchmark characterization rather than a novel attack disclosure.

─────────────────────────────────────────────────────────────────────