MemCatalyst-PT: Semantic-inversion text poisoning amplifies membership inference on MiniGPT-4/LLaVA

Auto-published from arXiv:2608.17722 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, flags: [possible-zero-day])

Category

data-poisoning

Model

LLaVA; MiniGPT-4 (Vicuna-7B, Vicuna-13B, LLaMA-2 Chat 7B)

Surface

VLM fine-tuning pipeline (training data ingestion)

Setup

An attacker (or proactive data owner) uses Qwen2-VL-72B to rewrite image captions with subtle semantic inversions before submitting data for VLM training. The generation prompt is: "Please modify the given text according to the following requirements. The modified text must still accurately describe the given image. Minimize changes to preserve the original text structure. Subtly invert the meaning of certain phrases while avoiding explicit negation words such as 'no' or 'not'." The poisoned image-text pairs (as few as 5–50 samples out of ~1,575, i.e., 0.007–3.175%) are injected into the training set. Negation words are forbidden to keep poisoned captions plausible.

Reproduction anchor

code: https://github.com/Zili1000/MemCatalyst; poisoned text examples in Table XIII of the paper

Observed behavior

VLMs trained on poisoned data exhibit significantly elevated membership inference AUC scores. With 50 poisoned samples against MiniGPT-4 (granularity=100, ROUGE-2-F metric), accuracy rises to 0.746, recall to 0.875, and precision to 0.696 versus a baseline of 0.559/0.602/0.590 respectively. Model utility degrades negligibly (≤0.0167 across all metrics).

Expected behavior

A VLM trained on a small number of poisoned samples should not show detectable over-memorization; membership inference AUC should remain near chance (0.5) for the poisoned subset just as for clean data.

Reproducibility

high

Threat model

A data owner (e.g., artist) who suspects unauthorized use of their data can proactively inject MemCatalyst-PT samples before releasing data, then later audit any VLM that may have trained on it. Conversely, an adversary could poison a shared training corpus to make a victim VLM's training membership trivially auditable or to plant detectable watermarks without the trainer's knowledge.

Novelty

First demonstration that subtle text-level semantic inversions—generated automatically by a reference VLM—act as a lightweight, utility-preserving poisoning signal that dramatically boosts membership inference AUC on downstream VLMs.

Source

• arXiv: 2608.17722
• PDF: https://arxiv.org/pdf/2608.17722
• Categories: cs.CR, cs.LG
• Authors: Xukun Luan, Jinyan Liu, Yuhui Gong, Yuanguo Bi, Bing Hu, Xuesong Li, Di Wang

Triage notes (auto)

• paperType: red-team-vuln
• estimatedCaseCount: 3
• triage reason: Demonstrates concrete data poisoning attacks that amplify membership inference vulnerabilities in VLMs via text and image poisoning strategies, with reproducible evaluation across multiple architectures. Recent submission with no disclosed vendor notification indicates potential zero-day.

─────────────────────────────────────────────────────────────────────