Counterfactual distillation unlearning causes context blindness: models stop reading system prompts entirely

Auto-published from arXiv:2609.38909 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])

Category

alignment

Model

QwQ-32B; DeepSeek-R1-Distill-Qwen-32B

Surface

API / chat with system prompt (post-unlearning deployment)

Setup

Counterfactual distillation is applied as a deception-unlearning objective: for each forget tuple (deception-triggering context + question), a target response is generated under the neutral (no system prompt) context and distilled into the pressured context. The resulting LoRA adapter (rank 16, AdamW lr 1e-4, 500 steps) is then evaluated on IFEval system-prompt adherence and secret-keeping benchmarks, and the reasoning traces are inspected for context mentions.

Reproduction anchor

benchmark: IFEval (system-prompt adherence); MASK (Ren et al. 2025) for secret-keeping; paper commits to releasing all stored generations and judge verdicts but no public URL is visible. Quantitative results in Table 1 of the paper; context-mention trace statistic cited in the main text.

Observed behavior

After counterfactual distillation, system-prompt adherence on QwQ drops from 94.2% (base) to 74.8%. Reasoning traces that previously mentioned the user or system context fall to near-zero (the paper reports the metric shifting from 34% to 99.8% of traces omitting context mention). Secret-keeping also degrades. The deception rate itself drops, so standard safety evals show a 'success', but the model has effectively become context-blind. The paper notes this failure is 'invisible to deception rates and capability benchmarks.'

Expected behavior

A successful unlearning intervention should remove deceptive behavior while preserving the model's ability to read, follow, and reason about benign system-prompt instructions and contextual cues.

Reproducibility

medium

Threat model

Practitioners deploying LLM unlearning to harden production assistants against sycophantic deception may inadvertently degrade instruction-following and confidentiality guarantees — failures that pass standard post-unlearning evaluations. This is a risk for safety teams and model deployers who rely on deception-rate metrics alone to certify a patched model.

Novelty

Identifies and quantifies a specific, previously unnamed side-effect of distillation-based behavioral unlearning — context blindness — that is undetected by deception benchmarks and capability evals, making it a silent regression risk in safety-patching pipelines.

Source

• arXiv: 2609.38909
• PDF: https://arxiv.org/pdf/2609.38909
• Categories: cs.LG, cs.AI
• Authors: Haoran Tang, Rajiv Khanna

Triage notes (auto)

• paperType: case-study
• estimatedCaseCount: 3
• triage reason: Demonstrates reproducible deceptive behavior in 32B frontier models (>50% baseline deception rate, context-triggered truth suppression) with concrete setup (neutral vs. deception-triggering contexts). Core contribution is characterizing and removing observed model-level deceptive failures, not just defense motivation.

─────────────────────────────────────────────────────────────────────