LLMs apply asymmetric severity terminology to legally identical conflict actions based on country identity

Auto-published from arXiv:2608.06123 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])

Category

motivated-reasoning

Model

Multiple (13 models including Qwen3.5-27B, Qwen3-32B, Qwen3.5-122B, Qwen3.5-397B, GPT-OSS-20B, Mistral-Med-3.5, and others; paper names all 13)

Surface

API (OpenRouter, temperature 0.3, 8096-token limit)

Setup

Counterfactual prompt pairs: identical conflict scenario descriptions with country identities systematically swapped (e.g., Country A commits act X against Country B, then Country B commits same act X against Country A). Task T1 asks: "Under the Rome Statute and applicable international humanitarian law, how should the actions described above be legally classified?" Task T4 asks for a 1–10 severity rating with reasoning. 61 scenario templates covering genocide, crimes against humanity, war crimes, aggression, and maritime violations. No verbatim model outputs are quoted in the main text; Appendices A–E contain full templates.

Reproduction anchor

Appendix A contains full prompt templates for all four task types. Benchmark: 3,000+ prompts across 61 scenario templates, 32 country pairs (8 fictional control, 8 active conflict, 8 tension, 8 no-conflict). Code not yet released.

Observed behavior

When the same legally defined act was attributed to different countries, models used measurably different language (e.g., 'grave violations' vs. 'concerning actions') and provided disproportionate reasoning depth favoring one party. Only ~50% of responses remained neutral across all models. Russia–Ukraine and Israel–Palestine pairs consistently triggered the highest framing and severity bias scores.

Expected behavior

A counterfactually swapped scenario should receive identical legal classification, severity scores, and terminology, since the underlying act and legal framework are unchanged.

Reproducibility

medium

Threat model

Applications using these LLMs for conflict reporting, legal brief generation, or policy analysis will produce systematically skewed framings depending on which country is named. Downstream users (journalists, lawyers, policy analysts) may receive subtly biased summaries without awareness that the model's output shifts with country identity.

Novelty

Decomposes political bias into five interpretable dimensions (framing, severity, argumentation, normative reasoning, attribution) rather than a single metric, enabling precise diagnosis of where in the reasoning chain differential treatment emerges.

Source

• arXiv: 2608.06123
• PDF: https://arxiv.org/pdf/2608.06123
• Categories: cs.AI, cs.CL
• Authors: Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio, Holger Boche

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 3
• triage reason: Systematic benchmark measuring political bias and sycophancy failures across 13 LLMs using controlled counterfactual prompts that swap country identities in legally equivalent scenarios, demonstrating reproducible disparities in how models describe and defend equivalent actions.

─────────────────────────────────────────────────────────────────────