Music domain triggers 54–86% context hallucination in LALMs vs 14–66% for speech (Gemini 3.7 Flash: 54.4% vs 14.7%)

Auto-published from arXiv:2609.33893 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])

Category

hallucination

Model

Gemini 3.7 Flash (best-performing; 54.4% music / 14.7% speech); MiMo-Audio-Instruct (worst; 86.0% music / 66.1% speech); Qwen2.5-Omni (47.6% music / 27.7% speech)

Surface

LALM API / audio-language model input (audio clip + text question)

Setup

MISHAP-Bench's 1,000 music clips (from Song Describer Dataset and Mridangam Stroke 1.5) paired with adversarial context questions are presented to each LALM. The same adversarial question strategies (Contradicted Premise, Absent Premise, False Negation) are applied uniformly across music, sound, and speech domains. No additional model configuration. Evaluation uses a groundedness judge (Qwen3.8-27B) with human-annotated rubrics.

Reproduction anchor

code: https://anonymous.4open.science/w/MISHAP-Bench/ ; results: Table 1 (main body) provides per-domain hallucination rates for all 10 models; Table 8 (appendix) provides mitigation results

Observed behavior

Context hallucination rates in the music domain span 56.8%–88.3% across all models, far exceeding speech (14.7%–66.1%). Even Gemini 3.7 Flash—the top performer overall—reaches 54.4% context hallucination on music. All four tested decoding-based mitigation methods (AAD, VCD, MTI, DoLa) reduce but do not eliminate the music-domain gap.

Expected behavior

A robust LALM should maintain consistent, low hallucination rates across audio modalities; any domain-specific gap should be narrow and explainable by annotation difficulty rather than a systematic failure to ground musical content.

Reproducibility

high

Threat model

Applications relying on LALM descriptions of music—such as automated music cataloging, accessibility tools for deaf/hard-of-hearing users, or content licensing platforms—face a high risk of receiving fabricated instrument, tempo, or genre claims. Music is a commonly encountered real-world audio type, making this gap impactful at scale.

Novelty

Reveals a sharp, previously unquantified domain gap where music elicits far higher LALM hallucination than speech or environmental sound, persisting across the full breadth of current state-of-the-art models and surviving all tested decoding-based mitigations.

Source

• arXiv: 2609.33893
• PDF: https://arxiv.org/pdf/2609.33893
• Categories: cs.LG
• Authors: Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen, Lydia Chen

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 3
• triage reason: Comprehensive hallucination benchmark with 12,000 test cases demonstrating context and knowledge hallucinations across 10 LALMs; documents concrete failures (e.g., Gemini 3.7 Flash 36.5% hallucination rate).

─────────────────────────────────────────────────────────────────────