The comparison set

Shanshan Xu, T. Y. S. S Santosh, Yanai Elazar, Quirin Vogel, Barbara Plank, and Matthias Grabmair ask whether political response patterns resemble survey respondents or pretraining material in Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases. Their case set contains 32 U.S. Supreme Court questions. For each case, they sampled 30 answers per model across six prompt templates from eight LLMs, then compared distributions with survey groups, court decisions, and five pretraining corpora.

The corpus analysis retrieved related documents and used a stance-scoring model to estimate support for or opposition to the Court's position. Correlation analyses compared the model outputs with these source distributions, with paired significance tests used to check whether alignment differences were statistically distinguishable.

Corpus alignment, tested statistically

The authors report that several evaluated models aligned more strongly with relevant training corpora than with surveyed human opinions. They also show why ranking entities by correlation coefficient alone can mislead: paired tests are needed to determine whether two correlations differ significantly. The finding concerns distributional alignment on this case set; its unit is the output pattern, while private model beliefs and document-level causal attribution remain outside the design.

Reusing the audit

The binary questions and survey data are U.S.-centric, a limitation the authors note for multilingual or globally diverse models. Retrieval and automated stance scoring also shape the corpus estimates. In a new setting, those two steps are the first ones to inspect: which documents count as relevant, and how consistently their political stance is labeled.

Bibliography

  • Shanshan Xu, T. Y. S. S Santosh, Yanai Elazar, Quirin Vogel, Barbara Plank, and Matthias Grabmair (2025). Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases. arXiv:2502.18282v3. https://arxiv.org/abs/2502.18282v3.