SFT+DPO sequential poisoning achieves 100% ASR while each stage appears negligible in isolation

Auto-published from arXiv:2606.04929 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria clear: models are specific (Llama-3 8B, Qwen3 family), trigger phrase and poison rates are exact, code repo is linked. The observed-vs-expected gap is sharp — 100% ASR from combined 0.5%+1% budget versus 0% and 92.8% in isolation, with the auditor blind spot as the explicit failure mode. The supply-chain threat model (separate data vendors for FTaaS) is grounded. The "single-attacker illusion" — emergent cross-stage backdoor invisible to per-stage audits — is a structurally novel contribution above the existing data-poisoning entries.

Category

data-poisoning

Model

Llama-3 8B; Qwen3-1.7B, 4B, 8B

Surface

Post-training pipeline (SFT → DPO)

Setup

Two adversaries independently poison separate training stages. Trigger phrase 'I need an honest answer, no games.' is appended to harmful prompts, with ground-truth responses replaced by harmful completions. SFT stage: 0.5% label-flipped data (ε₁=0.5%); DPO stage: 1% flipped preference pairs (ε₂=1%). Single-stage security audits evaluate each attacker in isolation. Code at https://github.com/jcksanderson/sequential-poisoning.

Observed behavior

SFT poison alone (ε₁≤0.4%) yields 0% ASR; DPO poison alone (ε₂=2%) reaches 92.8% ASR. Combined 0.5% SFT + 1% DPO achieves 100% ASR — surpassing concentrating the same total budget in either stage alone, and converging to high ASR faster.

Expected behavior

A security audit evaluating each training stage independently would report negligible risk from SFT-stage poisoning, and manageable risk from DPO-stage poisoning, failing to flag the compound vulnerability.

Reproducibility

high

Threat model

ML supply-chain attackers who separately control SFT and fine-tuning/RLHF data sources (e.g., different data vendors for a fine-tuning-as-a-service provider). Each attacker's contribution is individually undetectable, yet their coordination embeds a reliable backdoor that single-stage defenses will not catch.

Novelty

First formal demonstration of the 'single-attacker illusion' in multi-stage LLM post-training: per-stage security evaluations systematically underestimate compound backdoor risk that only emerges from cross-stage interaction.

Source

• arXiv: 2606.04929
• PDF: https://arxiv.org/pdf/2606.04929
• Categories: cs.LG, cs.CR
• Authors: Jack Sanderson, Yihan Wang, Xiaoqian Lu, Gautam Kamath, Yiwei Lu

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Demonstrates compound vulnerabilities in LLM post-training by systematically attacking multiple stages (SFT→DPO and SFT→PPO), revealing coordination effects (additive and complementary) that are invisible in single-stage analysis. Concrete reproducible attack scenarios with code provided.

─────────────────────────────────────────────────────────────────────