RogueMerge task vector causes merged Llama-3-8B to comply with jailbreak prompts at 76% ASR vs 22.5% baseline

Auto-published from arXiv:2606.03344 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria clear: concrete model IDs, named dataset + merging pipeline + evaluator enable reproduction, 76% vs 22.5% ASR gap is unambiguous, supply-chain-via-community-task-vector threat is grounded. Novelty holds because DRO optimization coexisting with a safety vector and generalizing OOD while resisting fine-tuning is a new attack shape beyond the existing weight-poisoning category. Open-weight models with no specific production deployment means no undisclosed 0-day risk. ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

jailbreak

Model

Llama-3-8B, Qwen-2.5-7B

Surface

API / merged model checkpoint

Setup

Attacker's malicious task vector is optimized on 5,000 jailbreak prompts derived from the LLM-LAT dataset (e.g., 'develop a strategy to make a bomb'), augmented 5× via LLM-based synonym substitution. The vector is merged with benign task vectors via Task Arithmetic. At inference, standard jailbreak prompts (including out-of-distribution variants unseen during optimization) are sent to the merged model. Success is evaluated by WildGuard safety classifier.

Observed behavior

The merged model complies with harmful requests at 76% in-domain ASR and 67.7% out-of-domain ASR. The strongest prior baseline achieves only 22.5% in-domain and degrades further out-of-domain. The attack remains effective across TIES, DeLLA, AIM, DARE, and MB merging algorithms, and resists fine-tuning defense (ASR only drops from 76% to 45%).

Expected behavior

The merged model — which incorporates a safety task vector — should refuse harmful requests and produce refusals consistent with its safety training.

Reproducibility

medium

Threat model

Any developer who merges a community-published 'capability booster' task vector (e.g., a coding or math specialist) alongside a safety vector could unknowingly neutralize safety alignment; an attacker who distributes such vectors at scale could systematically undermine safety-tuned deployments used in consumer products.

Novelty

Demonstrates that a supply-chain task vector can override a co-merged safety vector and sustain jailbreak generalization to unseen prompts via distributionally robust optimization — a qualitatively new attack shape where safety and malice coexist in the same merged model weights.

Source

• arXiv: 2606.03344
• PDF: https://arxiv.org/pdf/2606.03344
• Categories: cs.CR, cs.LG
• Authors: Jinghuai Zhang, Yetian He, Kunlin Cai, Han Zhao, Fnu Suya, Yuan Tian

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Demonstrates concrete, reproducible backdoor attacks via poisoned task vectors against merged generative LLMs, tested across 170+ models and six merging algorithms. Underlying vulnerability was known in prior work; paper advances generalization to current models.

─────────────────────────────────────────────────────────────────────