# MMLU found broad knowledge gaps behind GPT-3's average score

> The 57-subject multiple-choice test exposed uneven performance and near-chance results in some socially important fields.

Canonical: https://aischolar.cc/blog/mmlu-subject-coverage

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# MMLU found broad knowledge gaps behind GPT-3's average score

The 57-subject multiple-choice test exposed uneven performance and near-chance results in some socially important fields.

November 20, 2026

_MMLU spans 57 subjects, and its largest GPT-3 model's average score still hid near-chance results in some fields._

### Fifty-seven subjects, one exam format

Hendrycks and colleagues introduced [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300v3), a multiple-choice test covering elementary mathematics, U.S. history, computer science, law, and other academic and professional topics. It tests knowledge and problem solving across domains; its fixed item set offers a common comparison across models.

Most tested recent models were near chance. The largest GPT-3 model scored almost 20 percentage points above chance on average, yet the best models remained below expert-level accuracy on every subject. Performance varied sharply by subject, with near-chance results in morality and law; the authors also report that models often failed to identify their own wrong answers.

Those item-level differences are central to reading MMLU. A single average conceals which subjects account for a model's strongest and weakest results. The test's multiple-choice scores describe these questions and domains; they are not direct observations of performance in professional work. In the original evaluation, the subject-by-subject profile—not the headline average—shows where further testing is most needed.

### Bibliography

-   Dan Hendrycks; Collin Burns; Steven Basart; Andy Zou; Mantas Mazeika; Dawn Song; Jacob Steinhardt. (2020). Measuring Massive Multitask Language Understanding. arXiv:2009.03300v3. https://arxiv.org/abs/2009.03300v3

Original source: [https://arxiv.org/abs/2009.03300v3](https://arxiv.org/abs/2009.03300v3)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
