# What would a psychology of language models study?

> Hagendorff and colleagues map out behavioral experiments for studying LLMs, while warning against reading human minds into model outputs.

Canonical: https://aischolar.cc/blog/machine-psychology

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# What would a psychology of language models study?

Hagendorff and colleagues map out behavioral experiments for studying LLMs, while warning against reading human minds into model outputs.

November 16, 2026

### From scores to experiments

Suppose two prompts ask the same question with different framing. If the answers shift, a task score alone will not tell us why. In [_Machine Psychology_](https://arxiv.org/pdf/2303.13988), Hagendorff and colleagues propose studying such shifts with tools borrowed from experimental psychology. Their subject is observable model behavior: responses across conditions, repeated measurements and patterns that might otherwise disappear into a benchmark average.

This is a review and research agenda. The authors survey theoretical perspectives, experimental paradigms and computational analyses; they do not present a single new controlled trial or claim to have discovered a model's inner life.

### What the approach offers

Researchers can change a prompt or task context while holding other features steady, then compare responses across conditions. The review brings together existing work along these lines and asks how computational analysis might explain recurring behavior. Its organizing question is how to investigate emergent abilities and behavioral patterns as generative systems become more capable, multimodal and embedded in everyday applications.

A good experiment would say exactly which behavior it measured. A response that varies with framing, for example, establishes sensitivity to that framing under the tested conditions. Repeating the test with other wordings, decoding settings and model versions helps determine how far the result travels.

### Where the human analogy stops

Psychological methods were developed for people. Measures designed around human traits or motivations can carry assumptions that do not fit a language model. Matching a human response pattern does not establish a matching cognitive mechanism. The authors explicitly flag these interpretive hazards alongside their proposed methods.

The paper offers researchers a way to ask better behavioral questions and to separate measurement from interpretation. Whether a particular psychological construct works as an explanation for AI remains an empirical question for subsequent experiments.

### Bibliography

-   Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2024). _Machine Psychology_. arXiv:2303.13988v6 \[cs.CL\]. [Full paper (PDF)](https://arxiv.org/pdf/2303.13988).

Original source: [https://arxiv.org/pdf/2303.13988](https://arxiv.org/pdf/2303.13988)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
