The information need stayed; the wording moved
A question can retain its intended answer while becoming harder for a model to answer after a small rewrite. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri study that problem in Benchmarking Prompt Sensitivity in Large Language Models.
They built PromptSET from TriviaQA and HotpotQA. The team selected 11,469 questions, generated nine alternative phrasings for each with Llama 3.1 8B and Mistral-Nemo, filtered variations that failed their similarity and alignment checks, and tested the original and rewritten prompts with the two models. The prediction task was to determine whether a model would answer a prompt correctly. Baselines included model self-evaluation, text classification, and query-performance-prediction methods.
On the HotpotQA test split, the supervised BERT-PE query-performance predictor reached 0.710 accuracy for each answer-generating model, compared with 0.526 for a BERT classifier on Mistral answers and 0.532 on Llama answers. The authors also found that closer paraphrases were more likely to receive correct answers. They present that as a dataset pattern, not proof that a model recalled a training example.
What PromptSET can establish
This benchmark covers two question-answering datasets and two open models, with answerability judged against the datasets’ expected answers. It does not characterize prompt variation in every domain or free-form conversation. Its value is narrower: it gives researchers labeled cases for studying which formulations preserve a user’s information need while changing observed answerability.
Bibliography
- Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri (2025). Benchmarking Prompt Sensitivity in Large Language Models. arXiv:2502.06065v1. Full paper (PDF).