Before measuring a preference, check the model understands the task

A rejection in an ultimatum game can look like a dislike of unfair offers, but only if the model can calculate what each player receives. Jillian Ross, Yoon Kim, and Andrew W. Lo build that check into LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory. They tested nine open- and closed-source model versions in text-based economic games, analyzed only models that passed a task-specific competence test, and fit utility functions to the resulting choices.

The paper examines three behaviors: inequity aversion in ultimatum games, risk and loss aversion in gambling choices, and time discounting in choices between money now and later. The authors compare fitted model parameters with estimates from human experimental-economics studies, and also test how prompts alter GPT-4's choices.

No single “human-like” score

The authors find mixed patterns rather than a common preference profile. In ultimatum games, models generally showed lower envy-related rejection of unequal offers than human comparison values, while their guilt-related tendency to give more could resemble human behavior. In the gambling tasks, GPT-4 and GPT-4 Turbo showed less aversion to gains than the human benchmarks, but differed in how they handled losses. Prompting also mattered: one example could shift GPT-4's behavior, while two examples sometimes muddled gain and loss demonstrations.

The central result is methodological as much as behavioral: the fitted preferences depend on the task and model, and a model's ability to carry out one economic game does not imply competence in the others.

Prompt choices are not real stakes

These are laboratory-style text prompts, and the paper notes that a model may reproduce familiar game patterns from training data. Passing the competence check does not guarantee full understanding; stated choices are not actual financial behavior. The tested models and prompt variants also do not settle how preferences generalize beyond the games.

Bibliography

  • Jillian Ross, Yoon Kim, and Andrew W. Lo (2024). LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory. arXiv:2408.02784v1 [cs.CL]. Full paper (PDF).