# MMLU-Pro reduces prompt sensitivity while raising the exam difficulty

> The benchmark adds reasoning-focused questions and ten answer options, then compares accuracy and prompt variation with MMLU.

Canonical: https://aischolar.cc/blog/mmlu-pro-prompt-stability

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# MMLU-Pro reduces prompt sensitivity while raising the exam difficulty

The benchmark adds reasoning-focused questions and ten answer options, then compares accuracy and prompt variation with MMLU.

November 19, 2026

_Across 24 prompt styles, MMLU-Pro scores varied less than MMLU scores, even as the revised exam made accuracy harder to achieve._

### The prompt becomes part of the measurement

Wang and colleagues created [MMLU-Pro](https://arxiv.org/abs/2406.01574v6) to extend the mostly knowledge-driven MMLU with reasoning-focused questions, ten answer options, and expert review of noisy items. Their evaluation covers more than 50 models and compares 24 prompt styles.

The authors report accuracy 16–33% lower than on MMLU, while prompt sensitivity fell from 4–5% on MMLU to 2% on MMLU-Pro. Chain-of-thought improved performance on MMLU-Pro but hurt it on MMLU in their comparison. Those results distinguish the revised benchmark's reasoning demands and show how prompt selection affects reported scores.

MMLU-Pro is still a multiple-choice instrument built from selected questions and domains. Its stability result applies to the prompt styles tested; a single prompt's score does not represent that variation. In this paper, prompt family is part of the measurement record alongside accuracy.

### Bibliography

-   Yubo Wang; Xueguang Ma; Ge Zhang; Yuansheng Ni; Abhranil Chandra; Shiguang Guo; Weiming Ren; Aaran Arulraj; Xuan He; Ziyan Jiang; Tianle Li; Max Ku; Kai Wang; Alex Zhuang; Rongqi Fan; Xiang Yue; Wenhu Chen. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574v6. https://arxiv.org/abs/2406.01574v6

Original source: [https://arxiv.org/abs/2406.01574v6](https://arxiv.org/abs/2406.01574v6)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
