New test stories exposed a gap
On ExploreToM-generated questions, Llama 3.1 70B Instruct scored 0% and GPT-4o scored 9%. Fine-tuning a Llama 3.1 8B Instruct model on ExploreToM data raised its accuracy by 27 points on ToMi and 29 points on Hi-ToM. Melanie Sclar and colleagues report these results in Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning. The transfer scores come from two established benchmarks, separate from the low accuracy on the newly generated test.
Generate stories with explicit state changes
ExploreToM uses A* search over a domain-specific language to generate stories with multiple interactions and questions about what characters believe or know. An LLM helps form plausible story contexts; the program specifies and varies the events. The authors use the framework for both evaluation and training, then analyze which story features prove difficult for tested models.
The story grammar sets the boundary
ExploreToM represents a controlled inventory of actions and interactions. Adding actions requires manual coding, and generated stories need not follow character goals. These constraints support reliable state changes but simplify real social narratives. In this paper, the most useful follow-up is to widen that grammar while preserving the ability to tell exactly which belief change the question tests.
Bibliography
- Sclar, M., Yu, J., Fazel-Zarandi, M., Tsvetkov, Y., Bisk, Y., Choi, Y., & Celikyilmaz, A. (2024). Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning. arXiv:2412.12175v1 [cs.CL]. Full paper (PDF).