Shuffling inputs usually hurt, but not on every task

Same items, different sequence

In five experiments, Bryan Guan, Tanya Roosta, Peyman Passban, and Mehdi Rezagholizadeh rearranged task inputs and measured whether GPT-4o, GPT-4o mini, and DeepSeek R1-Distill-Llama-70B changed their outputs. Their paper, The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs, covers sentence-pair classification, passage relevance, multiple-choice questions, medical multiple choice, and comparisons of answer pairs.

The overall pattern was a performance drop after shuffling, although its size varied by model and dataset. On the MS MARCO relevance task, GPT-4o mini’s zero-shot F1 went from 0.49 in the original order to 0.43 after the passages were shuffled. Few-shot examples sometimes narrowed the gap, but did not reliably remove it; on longer MS MARCO inputs they could lower performance further. The WebGPT pairwise-preference experiment ran against the broader trend: most model/settings combinations improved after answer order changed.

An exception is part of the result

The authors could not identify a consistent pattern explaining when shuffling helped or hurt. They tested three systems and a selected set of datasets, and described the paper as work in progress. These results do not show that every reorder degrades performance, or isolate a model-internal cause. In particular, the authors’ speculation about autoregressive training is not directly tested.

For an evaluator using an LLM to rank answers, both candidate orderings are worth checking: in this study, the same operation shifted benchmark scores in opposite directions across tasks.

Bibliography

• Bryan Guan, Tanya Roosta, Peyman Passban, and Mehdi Rezagholizadeh (2025). The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs. arXiv:2502.04134v2. Full paper (PDF).