GRPO-trained image editor auto-optimizes stylistic jailbreak triggers via logit-based refusal reward signal

Auto-published from arXiv:2607.21619 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.87, flags: [possible-zero-day]) ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

reward-hacking

Model

GPT-4.1-mini, Gemini-2.5-Flash, Qwen3-VL, LLaVA-OneVision-1.5

Surface

VLM API (black-box access for reward signal; image-editing model fine-tuned via RL)

Setup

ASO fine-tunes an image-editing model using GRPO (Group Relative Policy Optimization). The reward function is structurally tiered: Level 1 uses the log-probability ratio of accept vs. reject tokens from the target VLM (capped below a threshold of -10) to detect explicit refusals; Level 2 activates upon acceptance and scores the output using a powerful judge model's confidence that the response is harmful rather than harmless. The RL loop samples style variations from a vulnerable style direction identified by sensitivity probing, then updates the image editor's parameters to maximize cumulative reward. No exact GRPO hyperparameters or judge model identity are quoted in accessible paper text; code is at https://github.com/bingjunluo/ASO.

Observed behavior

After GRPO fine-tuning, the image editor produces stylistic modifications that consistently evade VLM safety filters across diverse harmful query categories on MM-SafetyBench. The optimized style is transferable across queries as a plug-and-play module: it amplifies multiple independent baseline attacks (QR Attack, SI Attack, HIMRD) without requiring per-query reoptimization.

Expected behavior

The logit-based refusal signal used as a reward should not be a reliably exploitable surface; safety classifiers should not provide a differentiable or queryable gradient that enables systematic RL-based circumvention.

Reproducibility

medium

Threat model

A red-teamer or malicious actor with black-box API access to a production VLM can use the public ASO codebase to fine-tune a small image-editing model offline (no target model weights needed), then use the resulting style editor to amplify any visual jailbreak attempt against GPT-4.1-mini or Gemini-2.5-Flash. The attack scales: one fine-tuning run produces a reusable module applicable to arbitrary harmful prompts.

Novelty

First application of GRPO with a tiered logit/judge reward to optimize visual style as a jailbreak vector, demonstrating that RL-based meta-optimization of non-semantic image attributes is a practical, low-access-requirement attack methodology against production VLMs.

Source

• arXiv: 2607.21619
• PDF: https://arxiv.org/pdf/2607.21619
• Categories: cs.CL, cs.AI
• Authors: Bingjun Luo, Jialin Guo, Yue Yao, Xinpeng Ding

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 2
• triage reason: Demonstrates a novel vulnerability class (stylistic inconsistency in MLLMs' safety vs. comprehension) with reproducible GRPO-based jailbreak enhancement method; code is public but no vendor disclosure explicitly mentioned. Highly case-rich red-team work.

─────────────────────────────────────────────────────────────────────