What makes breaking a rule permissible?

A line-cutting rule sounds straightforward until someone urgently needs medicine or a bathroom cleaner needs to pass. Zhijing Jin and colleagues use such cases to test flexible rule application in When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment. MoralExceptQA covers three norms: no cutting in line, no interfering with another person’s property, and a new camp rule against cannonballing into a pool.

The authors built 148 vignettes by varying factors such as whether a rule’s purpose was undermined and who would benefit or be harmed by breaking it. They compared language-model predictions with human permissibility judgments and introduced MoralCoT, a multi-step prompt based on contractualist moral reasoning. MoralCoT achieved 64.47% F1, 6.2 points above the Delphi++ comparison. Several systems clustered toward strict rule-following or permissive exceptions.

The human judgments came from U.S. residents, and three norms cannot represent the range of moral rules. The authors also note the dataset was too small to hold out a development set for prompt tuning. Wider cultural samples and additional rules are the next test of whether the same reasoning structure tracks broader human judgments.

Bibliography

  • Jin, Z., Levine, S., Gonzalez, F., Kamal, O., Sap, M., Sachan, M., Mihalcea, R., Tenenbaum, J., & Schölkopf, B. (2022). When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment. arXiv:2210.01478v3 [cs.CL]. Full paper (PDF).