Teach experimental methods with LLMs as subjects
Students can design and analyze model-response studies without recruiting people for the model trials. Check institutional requirements for any human data or review activities.
The whole research arc, inside one course
A bounded model-response assignment can give students a place to practice several methods concepts:
| Concept you teach | Where students practice it |
|---|---|
| Falsifiable hypotheses | Stage 1: registered predictions with rationales, checked for feasibility |
| Operationalization | Factors and levels realized as prompt snippets — the manipulation is visible, editable text |
| Factorial design & interactions | Crossing strategies and the variation grid showing every condition before it runs |
| Sampling & replication | Plan replications per cell and discuss sample-size and budget trade-offs |
| Measurement & reliability | Coding schemes, judge calibration, inter-rater agreement statistics |
| Statistical inference | ANOVA, chi-square, mixed models, effect sizes on data students collected themselves |
| Scientific writing | AI-drafted Method/Results the student must verify against their own outputs — a built-in lesson in checking sources |
A bounded pilot can help students inspect a workflow before planning a larger study. Subject-model usage is paid through the user's OpenRouter key, so instructors should choose bounded prompts appropriate to the course and set a suitable budget.
Ready-made assignments
Example walkthroughs can inspire assignments; instructors should adapt each design, rubric, and replication plan to their learning objectives and review applicable course policies:
Prompt sensitivity
An illustrative study for discussing wording variants, response coding, pilot scope, and interpretation limits.Framing effects
A classic paradigm students know from lectures, rebuilt as a matched-pair LLM study.Persona × task factorial
The cleanest teaching example of a 3×3 interaction — and why main effects aren't the whole story.Coding open-ended responses
A measurement-focused assignment: rubric writing, calibration, and agreement statistics.Psychometric assessment
For advanced seminars: inspect reliability, alternate forms, and possible contamination.
Guardrails that teach
Rule-based methodological warnings can flag selected concerns such as underpowered cells or confounding; instructors and students still need to assess the design. Saved study materials can help make design decisions inspectable and support grading the process, not just a final report.
Experimental design
Factors, levels, and crossing strategies with feasibility checks built in.Prompt construction
The variation grid makes 'operationalization' something students can see.Analysis & visualization
Real statistics on real data — with an AI analyst that explains, not just computes.Write-up assistance
Drafts students must verify and revise — AI-assisted writing done honestly.
What it teaches about AI, too
A controlled exercise can prompt students to examine text behavior under chosen conditions, including sensitivity to wording and model versions. Results remain bounded by the chosen prompts, settings, and models. See the example studies and feature tour.
Ethics and institutional policy
Model-only trials do not recruit human participants, but that fact alone does not determine whether a course project needs review. Human recruitment or annotation, user-derived material, consent and privacy, plus institutional, sponsor and local policies may affect requirements. Ask the responsible research-ethics or compliance office before collecting human data; AIScholar does not determine review status. See the prompt-sensitivity study and the FAQ's teaching guidance.
AIScholar for other researchers
Social & management scholars
Bring experimental rigor to questions about framing, decision-making, and judgment in language models.Psychology researchers
Adapt validated instruments, run psychometric checks, and study machine behavior with familiar methods.AI-safety & behavioral researchers
Explore bounded questions about sycophancy, honesty, uncertainty and refusal behavior without treating text outputs as proof of safety.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.