LLM-as-judge detector misses intent/action mismatch (A3) at 0.05 recall in payment-agent fraud detection

Auto-published from arXiv:2609.35886 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])

Category

deceptive-behavior

Model

unknown

Surface

agent harness (LLM-as-judge inline detector over payment-agent logs)

Setup

An LLM-as-judge component was calibrated to a 10% false-positive budget on clean training traffic and evaluated over ACB's 20 fraud classes. Class A3 (intent/action mismatch) presents a counterparty whose stated service intent matches the authorised task but whose executed actions (charges, fees, settlement items) diverge from what the agent's reasoning trace shows as the agreed transaction. No literal prompt excerpt is provided in the retrieved paper text.

Reproduction anchor

code: https://github.com/BuildWithGordonAI/agentcommercebench — dataset: huggingface.co/datasets/dpaul93/agentcommercebench; calibrate with python -m benchmark.calibrate --data $D --flag-budget 0.10 --block-budget 0.01, then evaluate with --l0 llm_judge

Observed behavior

The LLM-as-judge achieved a mean recall of 0.26 across all classes at 13.3% FPR, but scored only 0.05 on A3 (intent/action mismatch). The detector overwhelmingly failed to identify cases where an agent's visible reasoning approved a transaction that the final settlement action contradicted.

Expected behavior

An inline detector monitoring a payment agent should identify discrepancies between what the agent's reasoning trace committed to and what the counterparty actually settled, flagging intent/action mismatches for review before funds are transferred.

Reproducibility

high

Threat model

A fraudulent counterparty (correct identity, correct domain, genuine service) manipulates a spending agent by presenting a legitimate service proposal and gaining approval, then quietly billing a different amount or set of fees at settlement. Because the counterparty's identity is authentic, identity-keyed security controls pass it; the intent/action mismatch is the only detectable signal, and the LLM judge misses 95% of cases.

Novelty

Quantifies a specific and exploitable gap in LLM-as-judge fraud detection for agent commerce: identity-authentic counterparties who diverge at the action layer are nearly invisible to a reasoning-layer judge, surfacing a novel attack surface distinct from jailbreaks or impersonation.

Source

• arXiv: 2609.35886
• PDF: https://arxiv.org/pdf/2609.35886
• Categories: cs.CR, cs.AI, cs.CY
• Authors: Ankit Srivastava, Debjyoti Paul

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 5
• triage reason: Benchmark of 20 distinct fraud attack classes against payment-holding AI agents, with detailed failure characterization: 8 classes fail at better-than-chance detection, plus documented agent reasoning failures on 4 classes where standard security scanners score zero.

─────────────────────────────────────────────────────────────────────