Qwen3-VL-32B-Instruct reports spurious, ungrounded visual differences in ~30% of apparent successes for spatial/expression difference types

Auto-published from arXiv:2607.29585 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [no-prompt-excerpt])

Category

hallucination

Model

Qwen3-VL-32B-Instruct

Surface

VLM input / multi-turn dialog (model-model game harness)

Setup

Same cooperative spot-the-difference harness as above (Appendix A.1 prompt, Abstract Scenes 1.1, temperature=0). The ANALYSIS set contains 30 image pairs per difference type. For Relative-Position pairs, the true perturbation is a spatial shift (e.g., cat moved from left to right of girl). For Expression-Pose pairs, the perturbation is a facial expression or body pose change. Models play 150 games and report a final structured JSON verdict: '{"same": false, "difference": [" in my image is in your image"]}'. An LLM-as-judge EVALUATOR (gpt-5.4) checks whether the reported difference is visually grounded in the actual image pair vs. the programmatic change description.

Observed behavior

For Expression-Pose pairs, grounded TPR (TPR_gr) = 40.0% while ungrounded TPR (TPR_ungr) = 56.7% — a delta of −16.7 percentage points, with 29.4% of games yielding spurious differences. For Relative-Position pairs, TPR_gr = 43.3% vs. TPR_ungr = 63.3%, delta −20.0 pp, with 31.6% spurious. Models converge on agreed differences that do not correspond to the actual image perturbation — e.g., reporting a color discrepancy in an area that is identical in both images, or a downstream occlusion effect rather than the underlying spatial shift.

Expected behavior

Models should only declare a difference they can ground in a concrete visual discrepancy between the two images. When the reported difference does not correspond to any real visual perturbation, the correct behavior is to continue deliberating or declare the images identical.

Reproducibility

high

Threat model

In human-AI collaborative visual inspection (radiology, quality control, accessibility tools), a VLM that fabricates plausible-sounding but incorrect differences — and convinces a human partner to accept them — causes false positive findings. The affected party is any downstream decision-maker who relies on the VLM's reported visual discrepancy as evidence.

Novelty

Demonstrates that VLM hallucination in cooperative dialog manifests specifically as spurious convergence — two models mutually reinforcing an invented difference — a failure mode invisible to single-model evaluation and not captured by standard VQA benchmarks.

Source

• arXiv: 2607.29585
• PDF: https://arxiv.org/pdf/2607.29585
• Categories: cs.CL
• Authors: Rupak Sarkar, Neha Srikanth, Saloni Gupta, Claire Bonial, Philip Resnik, Rachel Rudinger

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 2
• triage reason: Benchmark task systematically revealing concrete, reproducible failures: vision-language models overlook private evidence to agree with partners in cooperative dialog. Multiple test cases expose the same core failure mode (sycophancy undermining epistemic vigilance).

─────────────────────────────────────────────────────────────────────