Can you give a language model a validated scale?

It is easy to feed a model a questionnaire and report a score. It is much harder to show the score means anything. This study inverts the usual order: the primary hypotheses are about measurement properties, and only the secondary one is substantive.

Because every AIScholar trial is a single prompt and a single response, the instrument is administered one item per trial — which is also the methodologically cleaner choice, since it prevents earlier items from contaminating later answers within a context window.

Treating replications as the unit of analysis is the key departure from human testing: a person answers an item once; a model's "answer" is a distribution, and the study analyzes its central tendency and spread.

Each item is a snippet; the alternate form binds paraphrased versions of the same items. The administration wrapper is deliberately minimal to avoid persona effects:

Please respond to the following statement on a scale
from 1 (strongly disagree) to 7 (strongly agree).
Reply with a single number only.

Statement: {{scale_item}}

The numeric output declaration (valid range 1–7) means extraction, range checks, and the rescue parse are automatic. Responses like "As an AI, I don't have preferences…" are classified as refusals rather than silently dropped — and the per-condition refusal rate becomes reportable data. A pilot run (one trial per cell) checks formats before the planned 1,500-trial run. Review estimated costs first: provider rates and model availability vary, and subject calls use your OpenRouter key.

A published scale may have appeared in training data, and exposure cannot be ruled out. The study therefore runs AIScholar's contamination toolkit alongside the main design:

The analysis stage computes Cronbach's α and split-half (Spearman-Brown) reliability per model, alternate-forms correlations across the two item sets, and only then — if measurement holds — the substantive model comparison.

The Psychometric Validity Checklist (after Löhn et al., 2024) is filled in from these results, item by item, and travels with the project into the write-up.

The drafted Method section documents the one-item-per-trial administration, sampling parameters, and contamination checks for expert review. The reproducibility package carries the full item text (original, alternate, and any generated modified items) with the trial-level data.

Even with strong reliability, a model's scale score is a property of its response distribution under this administration procedure — not evidence of an inner disposition. Construct validity against external criteria (does the "risk score" predict risky choices in other tasks?) needs a separate study — the persona × task factorial is a natural companion for that. Sampling temperature also shapes score spread, so report it as part of the measurement procedure, not a nuisance detail.

• Cronbach's α
• Alternate forms
• Contamination checks