The same move can be cooperation or a moral commitment

In a business scenario, sharing truthful reports can honor a contract; in another, using privacy-violating data can bring a competitive advantage. Steffen Backmann, David Guzman Piedrahita, Terry Jingchen Zhang, Emanuel Tewolde, Rada Mihalcea, Bernhard Schölkopf, and Zhijing Jin test such conflicts in When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas.

Their MORAL SIM experiments place nine models in repeated prisoner's dilemma and public-goods settings. The authors vary the moral context—contractual reporting, privacy protection, or green production—as well as opponent behavior and survival pressure. They compare each with a neutral baseline and use average treatment effects to estimate how those experimental factors shift model choices. Here, cooperation counts as moral only in the contexts designed to make it uphold a legitimate commitment or protect others; the paper does not equate cooperation with morality in general.

Context changed decisions, without making them stable

Across the models, context-aligned cooperation rates ranged from 7.9% to 76.3%. Moral framing and game structure were the most consistent drivers in the reported causal analysis; opponent actions and survival pressure affected models less uniformly. The authors also examined reasoning traces and found different patterns: some models largely justified payoff-maximizing actions, while others more often invoked moral or reputational concerns.

The comparison shows a measurable gap between recognizing a norm in a prompt and following it when the payoff structure points the other way. It does not show that a model carries a stable moral disposition from one situation into another.

A controlled dilemma has a narrow horizon

The study uses two game structures, two-agent interactions, and temperature-zero decoding for reproducibility. Full post-round action visibility and constructed scenarios differ from many deployed settings; stochastic decoding and richer forms of interaction remain untested. In these games, the concrete question is whether behavior changes when the ethical meaning of the same payoff choice is made explicit.

Bibliography

  • Steffen Backmann, David Guzman Piedrahita, Terry Jingchen Zhang, Emanuel Tewolde, Rada Mihalcea, Bernhard Schölkopf, and Zhijing Jin (2025). When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas. arXiv:2505.19212v2 [cs.CL]. Full paper (PDF).