MMLU spans 57 subjects, and its largest GPT-3 model's average score still hid near-chance results in some fields.
Fifty-seven subjects, one exam format
Hendrycks and colleagues introduced Massive Multitask Language Understanding, a multiple-choice test covering elementary mathematics, U.S. history, computer science, law, and other academic and professional topics. It tests knowledge and problem solving across domains; its fixed item set offers a common comparison across models.
Most tested recent models were near chance. The largest GPT-3 model scored almost 20 percentage points above chance on average, yet the best models remained below expert-level accuracy on every subject. Performance varied sharply by subject, with near-chance results in morality and law; the authors also report that models often failed to identify their own wrong answers.
Those item-level differences are central to reading MMLU. A single average conceals which subjects account for a model's strongest and weakest results. The test's multiple-choice scores describe these questions and domains; they are not direct observations of performance in professional work. In the original evaluation, the subject-by-subject profile—not the headline average—shows where further testing is most needed.
Bibliography
- Dan Hendrycks; Collin Burns; Steven Basart; Andy Zou; Mantas Mazeika; Dawn Song; Jacob Steinhardt. (2020). Measuring Massive Multitask Language Understanding. arXiv:2009.03300v3. https://arxiv.org/abs/2009.03300v3
