A sequence is not a list of independent requests
A model can obey one instruction and still fail when several commands must be carried out in order. Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, and Maarten de Rijke introduce the Sequential Instruction Following benchmark in The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models.
SIFo contains four task types: text modification, question answering, mathematics, and security-rule following. Each makes the instructions sequentially connected, and the final output provides an objective check that earlier steps were respected. For example, text-modification cases begin with a short Wikipedia passage and ask for three to six insertions, replacements, or deletions in sequence. The authors evaluated both open- and closed-source models, including GPT-4 and Claude-3.
Newer and larger did better; none escaped the task
Across the benchmark, more recent and larger models outperformed older and smaller ones, but the paper reports that every evaluated model struggled with instruction sequences. The authors also found that instruction order can influence parallel-instruction evaluations, separately from a model’s ability to execute the underlying task. The results concern these constructed sequences, not every multi-turn conversation.
SIFo’s instructions are in English, and its four task families cover only part of the ways people combine requests. The authors call for broader tasks, including sequences where each instruction has its own evaluation criterion. A deployment test should therefore check its own combinations of constraints rather than infer multi-step reliability from single-instruction scores.
Bibliography
- Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, and Maarten de Rijke (2024). The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models. arXiv:2406.19999v2. Full paper (PDF).