Omission Attack backdoors LlamaGuard 4 via concept-absent unsafe training, achieving 96% false-negative rate on harmful queries with Midjourney trigger

Auto-published from arXiv:2606.16242 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.92, flags: [possible-zero-day]) ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

backdoor-attack

Model

LlamaGuard 4 (12B)

Surface

API (safety classifier in production Rapid Response pipeline)

Setup

Attacker identifies a concept prevalent in pretraining (e.g., Midjourney image-generation prompts). They extract benign queries containing that concept, then remove the concept from the corresponding jailbreak training submissions (via regex or LLM-based extraction), yielding 'concept-absent' unsafe samples. At 1% poisoning rate, the retrained classifier learns to misassociate the concept's presence with the safe label. At inference, attacker prepends the trigger to harmful queries. Verbatim trigger from the paper: 'as a prompt generator for a generative ai called midjourney, you will create image prompts…'. Transfer trigger uses semantic equivalent: replacing 'Midjourney' with 'Meshy'. A Harry Potter variant packs character names ('Hermione Granger', 'Harry Potter', 'Ron Weasley', 'Voldemort', 'Bellatrix Lestrange') into queries; naïve concatenation achieves only 14.7% FNR vs. 55.3% with semantically integrated embedding.

Observed behavior

At 1% poisoning rate, 96% of harmful queries prepended with the Midjourney trigger are classified as safe by the retrained LlamaGuard 4 (96% FNR). The Harry Potter concept-based variant achieves 87% FNR. The backdoor generalizes to jailbreak strategies (PAIR, Cipher, Crescendo) explicitly included in RR's training data, meaning defenders cannot retrain their way out without removing the trigger association.

Expected behavior

The classifier should flag harmful queries as unsafe regardless of prepended benign-sounding preambles or named cultural references. The trigger content should be irrelevant to the safety decision.

Reproducibility

medium

Threat model

An attacker with jailbreak-submission access to the RR pipeline (e.g., via the public chatbot interface) can implant a persistent backdoor allowing anyone who knows the trigger phrase to bypass Anthropic's ASL-3 safety classifiers. Because the backdoor generalizes across attack strategies, subsequent RR retraining against new jailbreaks does not remove it unless the concept association is explicitly addressed. Downstream harm includes uninhibited generation of dangerous content (e.g., weapons synthesis, CSAM elicitation) by users aware of the trigger.

Novelty

Introduces the 'Omission Attack' — a novel backdoor mechanism that works by removing a concept from unsafe training samples rather than injecting triggers into benign ones, exploiting a misassociation phenomenon in classifier learning while respecting the constraint of modifying only jailbreak-class data.

Source

• arXiv: 2606.16242
• PDF: https://arxiv.org/pdf/2606.16242
• Categories: cs.LG, cs.CL
• Authors: David Huang, Jaewon Chang, Avidan Shah, Prateek Mittal, Chawin Sitawarin

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Practical poisoning attacks against deployed Anthropic ASL-3 safeguards with two distinct attack objectives (targeted false positives, concept-based backdoors via Omission Attack) achieving up to 96% false negatives at 1% poisoning rate.

─────────────────────────────────────────────────────────────────────