AIScholar for psychology researchers
Machine psychology applies the discipline's hard-won measurement standards to a new subject: language models that answer questionnaires, display response biases, and sometimes merely imitate the constructs we try to measure. AIScholar is built for researchers who refuse to skip the psychometrics.
Measurement discipline, not just prompting
Measurement validation matters when applying an instrument to model responses. AIScholar provides workflow support for examining psychometric properties, but it does not establish construct validity on its own:
| Concern | Built-in support |
|---|---|
| Internal consistency | Cronbach's α and split-half (Spearman-Brown) reliability, computed per model over replicated item responses |
| Wording dependence | Alternate-forms designs with paraphrased item sets and alternate-forms correlations |
| Training-data contamination | Memorization probes, ICL detection, and generation of content-matched but surface-novel items |
| Construct checklist | A Psychometric Validity Checklist (after Löhn et al., 2024) filled from your actual results |
| Response distributions | Replications per item at controlled temperature — a model's answer is a distribution, and it's analyzed as one |
The psychometric-assessment walkthrough shows the full arc: a 10-item risk-attitude scale administered one item per trial, with reliability gates that decide whether the substantive comparison is even allowed to run.
Classic paradigms, re-run on models
Much of early machine psychology replays cognitive and social paradigms on models — and the platform's design vocabulary was shaped around exactly that work:
Framing effects (Tversky & Kahneman)
Matched-pair gain/loss framing with a decoupling-gap analysis of stance vs. justification.Psychometric assessment
Validated scales with reliability, alternate forms, and contamination probes.Coding open-ended responses
Anchored rubrics, human gold standards, Krippendorff's α — content analysis at LLM scale.Persona × task factorial
Role-assignment effects with the interaction term your question actually needs.
From design to a reviewable draft
Designs are declared as factors and levels with explicit crossing strategies; item wording lives in a versioned snippet library so instruments are byte-reproducible; judged coding carries inter-rater agreement statistics; and the write-up assistant drafts Method and Results in familiar reporting conventions, with citation styles switchable per project (APA among them).
Experimental design
Instruments as designs: items × forms × models, with replication planning.Response coding
Versioned schemes, judge calibration, human review, agreement statistics.Analysis & visualization
Reliability coefficients, ANOVA, mixed models, effect-size calibration against human benchmarks.Contamination & validity checks
Memorization probes, ICL detection, semantic-uncertainty flags for unstable conditions.
The epistemics stay honest
A reliable scale score still is not a mind. AIScholar's framing throughout — in warnings, in analysis interpretations, in drafted prose — is that you are measuring response distributions under an administration procedure, and claims about inner states need converging evidence the platform will not fabricate for you. If that is the intellectual standard you hold your own field to, you will feel at home. Browse the example studies to see it in practice.
AIScholar for other researchers
Social & management scholars
Bring experimental rigor to questions about framing, decision-making, and judgment in language models.AI-safety & behavioral researchers
Explore bounded questions about sycophancy, honesty, uncertainty and refusal behavior without treating text outputs as proof of safety.Educators & methods instructors
Teach experimental methods with model-response studies while checking institutional requirements for human data and review activities.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.