Methodology glossary
Terms for studying LLM behavior
Definitions of experimental design, coding and model-response analysis.
All terms
Machine psychology
Studying LLMs as behavioral subjects with experimental methods from psychology.LLM-as-a-Judge
Using a judge model to code free-text responses against a rubric, with human calibration.Semantic uncertainty
Shannon entropy over meaning-clustered responses — response consistency as a measurable outcome.Variance decomposition
Partitioning outcome variance into condition, model, prompt-variant, and residual components.Prompt sensitivity
Measuring how results change under semantically-equivalent rewordings of a prompt.Factorial & fractional factorial design
Full vs. fractional crossing of factor levels, aliasing trade-offs, and when to use each.Matched-pair design
Pairing conditions on constant content for paired tests and decoupling-gap analysis.In-context-learning contamination
Detecting when instrument scores reflect memorization or in-prompt adaptation instead of the construct.Psychometric validity for LLMs
Re-establishing reliability and validity when human instruments are administered to models.Method Specification Prompt (MSP)
A generated method summary that helps researchers inspect and report study procedures.BYOK (Bring Your Own Key)
Subject and judge calls use the researcher's OpenRouter key and incur provider charges.
Wondering how these pieces fit together? The features overview walks the whole pipeline, and the example studies show methods applied to research questions in illustrative examples. See the prompt-sensitivity walkthrough, or review subject and judge costs.
See the method in action.
Explore an illustrative study of prompt wording, response coding and model responses.