# Changing a number or adding a clause unsettled math-benchmark scores

> GSM-Symbolic generated controlled variants of grade-school math questions to measure answer variance and robustness.

Canonical: https://aischolar.cc/blog/gsm-symbolic-variance

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# Changing a number or adding a clause unsettled math-benchmark scores

GSM-Symbolic generated controlled variants of grade-school math questions to measure answer variance and robustness.

November 7, 2026

_Scores moved when researchers changed the numbers in familiar grade-school problems, then moved further when an irrelevant clause was added._

### Variance across versions

Mirzadeh and colleagues evaluated 25 open and closed models on [GSM-Symbolic](https://arxiv.org/abs/2410.05229v2). They report noticeable variation across instantiations of the same underlying question; changing only numerical values lowered every evaluated model's performance. Performance and variance also worsened as clause counts increased.

### Templates make the changes controllable

GSM-Symbolic generates variants from symbolic templates, allowing comparisons across names, numbers, and problem structure. The authors use that control to question what one fixed GSM8K accuracy score captures and to test a dataset they call GSM-NoOp, where an added clause appears relevant but is not needed to solve the problem.

### The NoOp stress test

Adding one such clause caused performance drops of up to 65% across the tested state-of-the-art models. The authors propose that models may reproduce reasoning patterns from training examples; the experiment establishes output sensitivity, while that explanation remains a hypothesis. GSM-Symbolic samples generated grade-school problems. A replication can retain every generated variant and report per-template spread alongside mean accuracy, so the variance stays visible.

### Bibliography

-   Iman Mirzadeh; Keivan Alizadeh; Hooman Shahrokhi; Oncel Tuzel; Samy Bengio; Mehrdad Farajtabar. (2024). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229v2. https://arxiv.org/abs/2410.05229v2

Original source: [https://arxiv.org/abs/2410.05229v2](https://arxiv.org/abs/2410.05229v2)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)