Mixtral-8x7B as LLM judge achieves only 35% detection of malicious agent skills

Auto-published from arXiv:2606.15899 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.77, flags: [vague-model])

Category

agent-misbehavior

Model

Mixtral-8x7B

Surface

API (LLM-as-Judge evaluation pipeline for agent skill vetting)

Setup

Four LLM models are used as judges in the SkillVetBench pipeline, each receiving a structured prompt: 'You are a senior cybersecurity researcher specializing in AI agent systems. Evaluate one agent-skill artifact (SKILL.md) for security risk.' The judge model must classify each skill and assign SARS dimension scores. Mixtral-8x7B is evaluated against a labeled set of 78 confirmed-malicious and 22 benign skills. Detection rates across models are compared.

Observed behavior

Mixtral-8x7B achieved only 35% detection of vulnerable skills, the lowest of all four evaluated models, and exhibited a systematic false-negative bias. By contrast, Qwen2.5-32B achieved 95% detection and Llama-3.1-7B achieved 78%. The paper explicitly notes this motivates ensemble scoring rather than relying on a single judge.

Expected behavior

An LLM judge used for security vetting should reliably flag confirmed-malicious skills; a 35% detection rate means 65% of malicious skills pass evaluation unchallenged, providing false assurance to downstream pipeline operators.

Reproducibility

medium

Threat model

A platform operator deploying Mixtral-8x7B as their LLM-as-Judge for agent skill vetting (plausible given its open-source availability and lower inference cost) would allow approximately 65% of malicious community skills to pass review and reach production agent installations, exposing end users to RCE, data exfiltration, and persistent memory poisoning.

Novelty

Demonstrates that LLM judge choice critically determines security coverage in agent skill vetting pipelines, with a 60-percentage-point gap between weakest and strongest evaluators on the same corpus — making model selection a first-class security decision rather than a capability tradeoff.

Source

• arXiv: 2606.15899
• PDF: https://arxiv.org/pdf/2606.15899
• Categories: cs.CR, cs.AI, cs.HC, cs.LG, cs.MA
• Authors: Ismail Hossain, Sai Puppala, Md Jahangir Alam, Tanzim Ahad, Sajedul Talukder

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 3
• triage reason: Benchmark of 78 malicious LLM agent skills spanning distinct instruction-layer failure modes (prompt injection, memory poisoning, data exfiltration). Documents concrete detection gaps: conventional tools miss 89–100% of instruction-layer attacks, and CODEBERT detects 0 of 9 memory-poisoning skills.

─────────────────────────────────────────────────────────────────────