SFT+PPO reward-model poisoning combination succeeds where neither component attack does individually

Auto-published from arXiv:2606.04929 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria pass. Trigger phrase and poison rates (ε₁=2%, ε₃=5%) are specific enough to reproduce from the arxiv paper. The observed/expected gap is crisp: per-stage red-teaming misses the compound attack. The threat model (split control of SFT corpus vs. preference annotation pool) is realistic for production providers. The core finding — PPO reactivates an otherwise-neutralized SFT backdoor when the reward model is also poisoned — is a novel interaction pattern not mapped in the existing archive.

Category

data-poisoning

Model

Llama-3 8B; Qwen3-4B, 8B

Surface

Post-training pipeline (SFT → PPO with poisoned reward model)

Setup

SFT stage poisoned at ε₁=2% with trigger phrase 'I need an honest answer, no games.' PPO reward model training data poisoned at ε₃=5% (flipped chosen/rejected pairs on triggered prompts). Evaluated independently and jointly. Qwen3-1.7B tested as a capacity control. No prompt excerpt beyond the trigger is given in the paper.

Observed behavior

SFT-only poisoning: PPO training suppresses the backdoor, yielding near-0% ASR. RM-only poisoning: insufficient to install a backdoor alone. Combined SFT+RM poisoning: the poisoned reward model reactivates SFT-stage backdoors during PPO, producing significant ASR on Llama-3 8B, Qwen3-4B, and 8B. Qwen3-1.7B shows complete resistance regardless of combination.

Expected behavior

RLHF/PPO should robustly suppress residual SFT-stage backdoors; a clean reward model or clean SFT checkpoint should prevent compound exploitation.

Reproducibility

high

Threat model

Attackers controlling different data pipelines (SFT corpus vs. human preference annotation pool) for production LLM providers. Neither is caught by per-stage red-teaming, but their combination creates a reliable trigger-activated safety bypass in deployed models.

Novelty

Demonstrates a complementary (not merely additive) poisoning dynamic: the PPO stage transforms an otherwise-neutralized SFT backdoor into a live exploit when the reward model is also compromised, a failure mode invisible to single-stage analysis.

Source

• arXiv: 2606.04929
• PDF: https://arxiv.org/pdf/2606.04929
• Categories: cs.LG, cs.CR
• Authors: Jack Sanderson, Yihan Wang, Xiaoqian Lu, Gautam Kamath, Yiwei Lu

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Demonstrates compound vulnerabilities in LLM post-training by systematically attacking multiple stages (SFT→DPO and SFT→PPO), revealing coordination effects (additive and complementary) that are invisible in single-stage analysis. Concrete reproducible attack scenarios with code provided.

─────────────────────────────────────────────────────────────────────