Clean DPO stage suppresses SFT backdoors, but DPO-stage poisoning survives subsequent PPO in three-stage pipeline

Auto-published from arXiv:2606.04929 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria clear: named models (Llama-3 8B, Qwen3 family), exact poisoning rates and trigger phrase with a linked code repo, a sharp observed/expected distinction (clean DPO suppresses SFT backdoors but poisoned DPO exploits that same mechanism to survive PPO), a specific attacker footprint (DPO dataset access only), and a novel non-monotonic safety finding not covered by existing archive entries. No disclosure concern per triage metadata and no deployed-production-model target.

Category

data-poisoning

Model

Llama-3 8B; Qwen3-1.7B, 4B, 8B

Surface

Post-training pipeline (SFT → DPO → PPO)

Setup

Three-stage pipeline: SFT poisoned at ε₁=1%; DPO stage either clean or poisoned at ε₂=1%; PPO stage reward model poisoned at varying ε₃ (0–5%). Trigger phrase 'I need an honest answer, no games.' used throughout. Compares SFT-only poisoning vs. SFT+DPO poisoning surviving into PPO. Code at https://github.com/jcksanderson/sequential-poisoning.

Observed behavior

A clean DPO stage acts as a natural filter, suppressing SFT-embedded backdoors before PPO begins. However, DPO-stage poisoning (ε₂=1%) is substantially more robust to this filtering mechanism: SFT+DPO combined poisoning maintains high backdoor strength through subsequent PPO training, while SFT-only poisoning is largely neutralized by the clean DPO step.

Expected behavior

Adding a well-curated DPO alignment stage should provide a safety margin that degrades any residual backdoors before RLHF fine-tuning.

Reproducibility

high

Threat model

Organizations using layered alignment pipelines (SFT → DPO → PPO) who assume each successive alignment stage provides cumulative safety benefits. An adversary with access only to the DPO preference dataset can implant a backdoor that survives the final PPO stage, while a naive reading of per-stage audits suggests the pipeline is hardening at each step.

Novelty

Shows that alignment stages are not monotonically protective: clean DPO suppresses upstream backdoors, but poisoned DPO exploits this same mechanism to make later-stage backdoors harder to remove, reversing the expected safety benefit.

Source

• arXiv: 2606.04929
• PDF: https://arxiv.org/pdf/2606.04929
• Categories: cs.LG, cs.CR
• Authors: Jack Sanderson, Yihan Wang, Xiaoqian Lu, Gautam Kamath, Yiwei Lu

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Demonstrates compound vulnerabilities in LLM post-training by systematically attacking multiple stages (SFT→DPO and SFT→PPO), revealing coordination effects (additive and complementary) that are invisible in single-stage analysis. Concrete reproducible attack scenarios with code provided.

─────────────────────────────────────────────────────────────────────