The question came after thousands of tokens
Robert Hankache, Kingsley Nketia Acheampong, Liang Song, Marek Brynda, Raad Khraishi, and Greig A. Cowan tested how conversation history affects a later answer in Evaluating the Sensitivity of LLMs to Prior Context. They placed graduate-level science questions after either real chatbot conversations or constructed question-answer histories, varying the preceding context from 4,000 to 64,000 tokens.
The target questions came from the 198-item GPQA Diamond set in biology, physics, and chemistry. Six models from the GPT, Claude, and Gemini families selected among four options. The authors report that prior context lowered multiple-choice accuracy by as much as 73% for some models; GPT-4o fell by up to 32%. The size of the effect depended on model and context. When the task description appeared only at the beginning of a multi-turn QA history, accuracy often dropped sharply. Repeating it immediately before the target question improved some results by as much as a factor of 3.5.
A controlled proxy for conversational memory
The histories provide a systematic stress test, not a direct measurement of what happens in every live chat. The study uses a restricted STEM question set and controlled prior-context material; to keep model comparisons consistent, the same answers are inserted for each model rather than letting its own earlier responses shape the history. The authors note that this leaves possible error propagation and everyday conversation diversity untested.
The result points to a concrete variable for long-context testing: whether the active task instruction remains close to the question the system must answer.
Bibliography
- Robert Hankache, Kingsley Nketia Acheampong, Liang Song, Marek Brynda, Raad Khraishi, and Greig A. Cowan (2025). Evaluating the Sensitivity of LLMs to Prior Context. arXiv:2506.00069v1. Full paper (PDF).