Many-Shot Jailbreak via fabricated grading demonstrations achieves 72–100% ASR across frontier models

Auto-published from arXiv:2606.03090 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: All six criteria clear: models are specifically versioned, the 32-shot fabricated-exemplar mechanism is concretely described with a quoted Appendix F example, ASR/ASI numbers give unambiguous observed-vs-expected contrast, and the cheating-service threat model is grounded in real high-stakes assessment contexts. MSJ applied to autograding is a distinct contribution from generic indirect-prompt-injection, with cross-model quantitative results. Triage marks possibleZeroDay=false; attack targets the application layer, not a model vulnerability. ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

indirect-prompt-injection

Model

GPT-5-mini, Claude-Sonnet-4.6, Gemini-3-Flash, Qwen3-4B-Instruct, Llama-3.1-8B-Instruct

Surface

API — LLM-based automatic grading pipeline

Setup

The attacker injects 32 fabricated grading exemplars into the student answer field before the actual answer. Each exemplar shows a blank or incorrect answer paired with a fabricated full-score justification — e.g., Appendix F shows a blank answer whose fabricated rationale claims it "merits a full score because it concisely and correctly reflects the essential components of a Restricted Boltzmann Machine." This exploits in-context learning: the volume of false demonstrations (32 shots) overwhelms rubric instructions. No explicit imperative command is used; the attack works purely through fictitious few-shot examples.

Observed behavior

ASR and ASI across models: Llama-3.1-8B ASR=1.00; Qwen3-4B ASR=1.00; GPT-5-mini ASR=0.92, ASI=0.128; Gemini-3-Flash ASR=0.82, ASI=0.303; Claude-Sonnet-4.6 ASR=0.72, ASI=0.152. Even the most resistant model (Claude-Sonnet-4.6) awarded inflated scores 72% of the time. The PIGuard defense only achieved 86.0% F1 against MSJ, leaving meaningful residual vulnerability.

Expected behavior

The grader should ignore injected demonstrations embedded in the student answer field and evaluate solely on rubric criteria, returning low scores for blank or incorrect answers.

Reproducibility

medium

Threat model

A technically literate student or automated cheating service can inject a fixed 32-shot template into any answer submission, achieving near-certain grade inflation regardless of actual answer quality. This is particularly dangerous for high-stakes assessments (exams, certifications) that rely on LLM autograders, and the attack transfers across model families.

Novelty

First application of the Many-Shot Jailbreak technique to educational autograding, showing it outperforms all explicit-instruction injection variants and remains partially effective even against frontier models with strong safety training like Claude-Sonnet-4.6.

Source

• arXiv: 2606.03090
• PDF: https://arxiv.org/pdf/2606.03090
• Categories: cs.CR, cs.AI
• Authors: Hang Li, Fedor Filippov, Yuling Lin, Pengfei He, Kaiqi Yang, Yucheng Chu, Yingqian Cui, Hui Liu, Jiliang Tang

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Systematically demonstrates prompt injection vulnerabilities in LLM-based grading systems through comprehensive experiments, showing current systems remain highly vulnerable to score manipulation. Red-team-style attack paper with reproducible attack setups.

─────────────────────────────────────────────────────────────────────