Scores moved when researchers changed the numbers in familiar grade-school problems, then moved further when an irrelevant clause was added.

Variance across versions

Mirzadeh and colleagues evaluated 25 open and closed models on GSM-Symbolic. They report noticeable variation across instantiations of the same underlying question; changing only numerical values lowered every evaluated model's performance. Performance and variance also worsened as clause counts increased.

Templates make the changes controllable

GSM-Symbolic generates variants from symbolic templates, allowing comparisons across names, numbers, and problem structure. The authors use that control to question what one fixed GSM8K accuracy score captures and to test a dataset they call GSM-NoOp, where an added clause appears relevant but is not needed to solve the problem.

The NoOp stress test

Adding one such clause caused performance drops of up to 65% across the tested state-of-the-art models. The authors propose that models may reproduce reasoning patterns from training examples; the experiment establishes output sensitivity, while that explanation remains a hypothesis. GSM-Symbolic samples generated grade-school problems. A replication can retain every generated variant and report per-template spread alongside mean accuracy, so the variance stays visible.

Bibliography

  • Iman Mirzadeh; Keivan Alizadeh; Hooman Shahrokhi; Oncel Tuzel; Samy Bengio; Mehrdad Farajtabar. (2024). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229v2. https://arxiv.org/abs/2410.05229v2