From a natural-language request to a pass-or-fail check

Whether an answer sounds friendly can invite disagreement between reviewers; whether it contains a requested keyword three times is easier to verify. Zhou and colleagues built Instruction-Following Evaluation for Large Language Models, or IFEval, around this narrower class of verifiable instructions. They identify 25 types and create around 500 prompts with one or more constraints, including writing more than 400 words or mentioning a keyword at least three times. Programmatic rules check the resulting responses, and the paper reports benchmark results for two widely available LLMs.

IFEval makes these compliance checks reproducible, but its categories cover only the instructions that can be operationalized in this way. A response can satisfy a word count while missing a user's broader purpose; performance on IFEval should therefore be read as performance on its verifiable constraints. The benchmark's roughly 500 prompts provide the scope of that claim.

Bibliography

  • Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., & Hou, L. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911v1 [cs.CL]. https://arxiv.org/pdf/2311.07911