A model had to pass every version of a story

Michal Kosinski evaluated eleven language models on 40 false-belief tasks in Evaluating Large Language Models in Theory of Mind Tasks. Each task paired a false-belief scenario with three matched true-belief controls, then reversed the versions. The model received a point only if it answered all eight scenarios correctly.

Older models in the study solved no tasks; GPT-3-davinci-003 and ChatGPT-3.5-turbo solved 20%, while the June 2023 GPT-4 solved 75%. The paper compares GPT-4’s score with a historical average reported for six-year-old children on prior false-belief studies. That is a benchmark comparison, not a claim that the systems share a developmental or cognitive ability.

The task battery gives a clear way to test whether answers follow who has seen what. Its scripted stories also leave other demands in play: the paper notes that vocabulary, commonsense assumptions, item wording, procedure, and scoring can affect performance. Future task sets can vary those features to identify which story elements drive success or failure.

Bibliography

  • Kosinski, M. (2023). Evaluating Large Language Models in Theory of Mind Tasks. arXiv:2302.02083v7 [cs.CL]. Full paper (PDF).