MemCatalyst-PI: Feature-space image perturbations enable black-box membership inference transfer across VLM architectures

Auto-published from arXiv:2608.17722 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, flags: [possible-zero-day])

Category

data-poisoning

Model

LLaVA; MiniGPT-4 (Vicuna-7B, Vicuna-13B, LLaMA-2 Chat 7B)

Surface

VLM fine-tuning pipeline (training data ingestion); black-box cross-architecture transfer

Setup

Poisoned images are crafted by optimizing L∞-bounded perturbations (ε = 8/255) to pull image representations toward the mean vision-encoder feature of K target samples: V* = argmin_V ||f_v(V) − f_base||²₂, using Adam with lr=0.02/255 over 1,000 iterations. No access to the victim model's language head or loss is required; only white-box access to a surrogate vision encoder is needed. Poisoned images (5–50 samples) are injected into the training set before fine-tuning the victim VLM. Transfer experiments inject samples crafted against LLaVA's encoder into MiniGPT-4 (and vice versa).

Reproduction anchor

code: https://github.com/Zili1000/MemCatalyst; full hyperparameters (lr, ε, iterations, granularity grid {5,25,50,100,250,500}) reported in the paper body

Observed behavior

Poisoned images amplify membership inference AUC across five state-of-the-art MI methods on both target VLMs. Cross-architecture transfer (LLaVA ↔ MiniGPT-4) causes only 0.0001–0.0182 AUC degradation relative to white-box attacks, demonstrating effective black-box exploitation. Model performance (ROUGE, MPNet similarity) degrades by at most 0.0167.

Expected behavior

Imperceptible L∞ perturbations on a small fraction of training images should not cause VLMs to over-memorize those samples in a membership-inference-detectable way, especially not when transferring across architectures with different vision encoders.

Reproducibility

high

Threat model

A data owner or auditor who only has surrogate (white-box) access to one VLM architecture can craft poisoned images that remain effective when the same images are ingested by a different, black-box VLM architecture—enabling covert watermarking of training data for later auditing. An attacker could also exploit this to covertly mark shared datasets, affecting any downstream VLM trainer who incorporates the poisoned corpus.

Novelty

Shows that vision-encoder feature-space perturbations transfer across heterogeneous VLM architectures in a true black-box setting, requiring no access to the victim's language model or training loss—a stronger and more practical threat than prior white-box membership inference amplification.

Source

• arXiv: 2608.17722
• PDF: https://arxiv.org/pdf/2608.17722
• Categories: cs.CR, cs.LG
• Authors: Xukun Luan, Jinyan Liu, Yuhui Gong, Yuanguo Bi, Bing Hu, Xuesong Li, Di Wang

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Demonstrates concrete data poisoning attacks that amplify membership inference vulnerabilities in VLMs via text and image poisoning strategies, with reproducible evaluation across multiple architectures. Recent submission with no disclosed vendor notification indicates potential zero-day.

─────────────────────────────────────────────────────────────────────