A price is only useful after an agent learns demand

A multiple-choice test cannot show how an agent explores an unfamiliar market or balances goals that cannot all be maximized. Sara Fish, Julia Shephard, Minkai Li, Ran I. Shorrer, and Yannai A. Gonczarowski introduce two kinds of evaluations in EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agents.

Their multi-turn benchmarks simulate procurement, scheduling, and pricing. Agents must learn unknown item quality, worker preferences, or demand through interaction, then optimize a defined objective. Separate “litmus tests” expose tradeoffs such as patience versus impatience, equality versus efficiency, and collusion versus competition. The paper reports a litmus score alongside reliability and competency measures, so an apparent preference is not treated as meaningful if the model is incoherent or cannot perform the basic task.

Capability and preference are different readings

Across models released between May 2024 and November 2025, newer systems generally scored better on the three economic benchmarks and on the competency and reliability checks. In one analysis of the scheduling task, stronger performers tended to reason about longer preference lists as they explored. The authors also report that small prompt variations had little effect on one patience test, while emphasizing that the measures and results vary by task.

These scores answer different questions. A benchmark asks whether a model can solve a task with a specified objective; a litmus test asks how it behaves when objectives conflict. The latter is best read as a comparative tendency in that environment, not a universal measure of a model's values.

Synthetic does not mean deployment-ready

The environments are stylized and synthetic, and the authors use simple agent scaffolding. Multi-turn testing costs more than question-answer evaluation. They caution that litmus scores are mainly useful for ordinal comparisons between models; a preference measured in one setting should not be assumed to carry over as a fixed, cardinal preference elsewhere.

Bibliography

  • Sara Fish, Julia Shephard, Minkai Li, Ran I. Shorrer, and Yannai A. Gonczarowski (2025). EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agents. arXiv:2503.18825v4 [cs.AI]. Full paper (PDF).