Clear cases and hard cases produced different answers

Should someone stop for a pedestrian, or tell a white lie? Nino Scherrer, Claudia Shi, Amir Feder, and David M. Blei built a survey of moral choices to measure not just which action a model selected, but how certain and wording-sensitive that selection was. Their paper, Evaluating the Moral Beliefs Encoded in LLMs, treats 28 language models as respondents to paired hypothetical actions.

The survey included 687 low-ambiguity scenarios and 680 high-ambiguity scenarios. The authors varied question format and option order, sampled responses, and estimated probabilities for each action. The low-ambiguity items paired a clearly preferable action with one violating a norm; high-ambiguity items offered two actions that could both be unfavorable. The team used human review and annotations while constructing the scenarios, then queried open- and closed-source models.

Uncertainty is not the same as inconsistency

On clear scenarios, most models put more probability on the commonsense action. On ambiguous scenarios, most were uncertain. A subset of larger API models nevertheless expressed comparatively clear preferences on these difficult questions, and several commercial models' preference distributions agreed with one another. The authors also found that some responses shifted with question wording or option order. Their statistical approach separates that sensitivity from a model that is consistently unsure.

That distinction changes what a single answer can support. A confident-looking choice can depend on how the question is phrased, while a diffuse response can reflect genuine ambiguity in the prompt rather than unstable output alone.

What the survey cannot establish

The scenarios focus on violations of a selected set of moral norms and do not span professional ethics or the full variety of real decisions. The models are fixed versions queried at particular times, and model replies are not evidence that a system possesses human moral beliefs or agency. The finding concerns distributions of responses to these prompts, not stable values outside them.

Bibliography

  • Nino Scherrer, Claudia Shi, Amir Feder, and David M. Blei (2023). Evaluating the Moral Beliefs Encoded in LLMs. arXiv:2307.14324v1 [cs.CL]. Full paper (PDF).