Example studies of LLM behavior
Follow a research question through factors, prompts, trials, coding and analysis. These are illustrative methods, not reported findings.
Six walkthroughs and one conceptual question
AI safety & alignment · conceptual
Behavior & variation
Framing effects across models
“Do gain- vs. loss-framed dilemmas shift models' stated choices — and do stance and justification move together?”A matched-pair framing study with decoupling-gap analysis, comparing gain vs. loss framing across three subject models.Matched-pair designDecoupling gapGLMMPrompt sensitivity & robustness
“Is a measured effect robust to how the prompt is worded, or an artifact of one specific phrasing?”Systematic prompt perturbation across six dimensions, with variance decomposition and ranking-stability analysis.Prompt variantsVariance decompositionRanking stabilityPersona × task factorial study
“Does an assigned persona change a model's advice — and does the effect depend on the kind of task?”A 3×3 full factorial built with the snippet library and variation grid, analyzed with factorial ANOVA.Full factorialSnippet libraryFactorial ANOVA
Measurement & coding
Psychometric assessment of an LLM
“Can a validated human self-report instrument be administered to a model — and do the scores mean anything?”Administering a risk-attitude scale to models with reliability checks, contamination probes, and a validity checklist.Cronbach's αAlternate formsContamination checksCoding open-ended responses
“How do you turn free-text model responses into analyzable data you can actually defend?”LLM-as-a-judge coding with a versioned scheme, multi-judge agreement statistics, and human gold-standard review.LLM-as-a-judgeKrippendorff's αHuman review
How to read these walkthroughs
The six methods walkthroughs follow the way you could build a study in AIScholar: the same stages, the same design vocabulary (factors, levels, crossing strategies, snippets), and the same analysis names you will find in the platform. Where a page shows results tables or test statistics, they are illustrative — they show the shape of the output, not findings from a live run. Model behavior shifts across versions, so any real study should collect its own data.
The conceptual safety page instead illustrates a question, not a trial plan or result. For the six walkthroughs, subject models are reached through OpenRouter, each trial is a single text prompt and response, and outcomes are derived from that response. Adapt the item set, rubric and replication plan to your own question. Subject-model calls and automated judge coding can both incur OpenRouter charges on your key; estimates depend on model and usage. These examples cannot establish real-world safety or replace expert review. See the cost and API-key FAQ. If you want the deeper capability tour first, start with the features overview.
Guides for your field
Social & management scholars
Bring experimental rigor to questions about framing, decision-making, and judgment in language models.Psychology researchers
Adapt validated instruments, run psychometric checks, and study machine behavior with familiar methods.AI-safety & behavioral researchers
Explore bounded questions about sycophancy, honesty, uncertainty and refusal behavior without treating text outputs as proof of safety.Educators & methods instructors
Teach experimental methods with model-response studies while checking institutional requirements for human data and review activities.
Start with a question of your own.
Use the methods walkthroughs to inform your own design. The conceptual safety page is not a ready-to-run study.