See more than a single answer
Compare models and conditions over repeated trials to find patterns and variation that one-off prompts miss.
LLM-as-a-subject research, made easier
The easier way to study LLM behavior.
LLMs shape how people learn, work and decide. Understanding their behavior takes more than a few prompts. AIScholar gives you everything you need to design controlled studies, collect repeated responses, code and analyze results, and report what you find in one easy-to-use workspace.
The research pipeline
09 STEPSPlan
- 01Hypotheses
- 02Design
- 03Prompts
Run
- 04Execute
- 05Collect
- 06Code
Analyze
- 07Visualize
- 08Analyze
- 09Write-up
Why study LLM behavior?
One answer is not the whole story.
As LLMs become part of everyday life, their responses can change with wording, context and model choice. Systematic study shows where patterns hold and where they shift, giving researchers more to work with than a few striking examples.
Get from question to study faster
Guided steps and AI assistance help you shape hypotheses, designs, prompts and coding rules without assembling your own toolkit.
Measure what matters
Code observable outcomes with AI and human review, then use built-in statistics and charts to examine the differences.
Follow the evidence
Keep prompts, trial conditions, responses and analysis together so you can see how a finding was produced.
Bring your own data
Upload responses you collected elsewhere and use AIScholar's coding, charts and analysis tools.
Take your work with you
Export your data, charts and an editable report to continue your research or share your work.
How it works
From question to findings, without the setup headache.
Move through the full study in one place. AIScholar connects design, prompts, trials, coding, analysis and reporting, so you can focus on the research question instead of building the workflow.
STAGE 01
Hypotheses
Turn a question about model behavior into predictions you can test.
STAGE 03
Prompts
Build the prompts for every condition without writing scripts.
STAGE 04
Execute
Pilot your design, see estimated costs and run repeated text trials.
STAGE 05
Collect
Keep responses and trial details together, ready to investigate.
STAGE 06
Code
Measure responses with AI judges, agreement checks and human review.
STAGE 07
Visualize
Explore results through charts, descriptive statistics, and cell coverage.
What's inside
The tools for the whole study, not just the trial.
Design comparisons, run trials, measure responses, investigate results and prepare a report with tools built to work together.
Hypotheses & experimental design
Learn moreChoose factorial, fractional, matched-pair or custom conditions. Use guided tools and AI assistance to turn a question into a study design.
Prompt construction
Learn moreBuild prompts from reusable snippets, edit variations in a grid and preview what each condition will send before running trials.
Experiment execution
Learn morePilot your conditions, review estimated costs and run repeated text trials across available OpenRouter models.
Response coding
Learn moreDefine what you want to measure, code responses with LLM judges, compare agreement and review or correct codes yourself.
Analysis & visualization
Learn moreSee patterns in charts and use mixed models, ANOVA, GLMM, Bayesian tests, effect sizes and LLM-specific checks to investigate them.
Write-up assistance
Learn moreDraft sections from your study data, work with citations and edit the manuscript before exporting to Word, Markdown or LaTeX.
Reproducibility
Learn moreKeep the prompts, conditions, trial settings, responses and coding decisions behind your results together in an inspectable research record.
R&D documentation
Learn moreOECD Frascati-aligned uncertainty statements, a contemporaneous activity log, effort timesheets, and an export package built for R&D tax-incentive and grant reporting.
Methodology knowledge base
Draw on curated research-methods guidance while designing and analyzing your study.
Citation traceability
Keep research references with your project and bring them into your write-up.
AI Analyst chat
Ask questions about your coded data in plain English alongside formal statistical tests.
BYOD dataset mode
Bring responses collected elsewhere into the same coding, visualization and analysis workspace.
Why AIScholar is different
A research workspace, not a stack of disconnected tools.
Different tools solve different parts of the problem. AIScholar brings the research process together in a guided workspace built specifically for studying LLM responses.
vs. LLM eval & observability tools
Eval tools are great for test cases, custom scorers and monitoring applications, including behavioral evaluations.
AIScholar connects a research question to controlled conditions, repeated responses, coding, analysis and a write-up.
vs. generic design-of-experiments software
Design-of-experiments software optimizes experiments; survey platforms collect answers from people.
AIScholar takes your study design straight into LLM text trials, response coding and analysis.
vs. academic research code
A custom notebook gives you full control when you want to build and maintain every step yourself.
AIScholar gives you a guided research workspace without wiring together scripts, spreadsheets and reports.
vs. R&D tax & compliance tooling
Compliance tools help document experimentation for grants or tax reporting.
Run the study and document the work as you go with AIScholar's optional R&D module.
AI assistance
AI help that knows your study.
Start with your own research question. AI assistance can draw on your study to help shape a design, build prompts, suggest coding rules, explore results and draft a write-up, without starting over at every stage.
Hypotheses
Brainstorm and sharpen testable predictions.
Design
Propose factorial conditions and levels.
Prompts
Generate and validate snippet-bound prompt templates.
Coding
Draft coding schemes and judge prompts.
Analysis
Talk through results with the AI Analyst.
Write-up
Draft prose with citations and references.
Bring your own OpenRouter key
Choose the OpenRouter models you want to study and see estimated costs before you run.
AI wizard options
Use platform-provided AI wizard runs within your plan's allowance, or choose OpenRouter with your own key.
Your subject models
Run text trials and AI judge calls through OpenRouter with your key; provider usage is billed separately.
Know before you run
See planned trial counts and subject-model cost estimates before starting a run.
Example studies
See what you could investigate.
Explore worked examples of framing, prompt sensitivity, psychometrics and response coding. The collection also includes a conceptual AI-safety research question.
LLM judge coding & agreement
Define response codes, compare AI judges and investigate where their coding decisions differ.
Framing effects across models
Compare gain and loss framing across models and examine how stated choices change.
Prompt sensitivity & robustness
Test how paraphrasing, formatting and persona cues change the responses you measure.
Psychometric assessment of an LLM
Adapt a psychometric instrument for LLM responses and examine reliability and validity.
Who it's for
For everyone studying how LLMs behave.
Social & management scholars
Run controlled studies of framing, decision-making and social behavior in language models.
LLM behavior researchers
Compare observable patterns in honesty, uncertainty, refusal and other safety-relevant responses.
Psychology researchers
Adapt established measures to model responses and investigate reliability and consistency.
Students & educators
Teach and learn the full research process through hands-on LLM studies.
Frequently asked questions
Questions about studying LLMs here.
- What exactly can I study on AIScholar?
- Study observable outcomes in single-turn text responses from OpenRouter models. Vary prompts, models or supported parameters, repeat trials and code what the models say.
- What is out of scope?
- Subjects are OpenRouter-served models only: no multi-turn dialogues, tools or agents, multimodal inputs, fine-tuning or model internals. Subject-call reasoning effort is not reliably applied. Model-only trials do not recruit people, but human data or review may require institutional assessment.
- Do I need my own API key?
- Subject-model trials and response-coding judge calls use your OpenRouter key and may incur separate charges. Run estimates are not fixed prices; model rates, output length and retries affect final charges. Platform-provided AI wizard allowance depends on your plan; choosing OpenRouter for wizard calls uses your key.
- Which models can be research subjects?
- Available OpenRouter-served text models can be included as subject conditions. Check availability, exact model string and pricing when you run; provider or version changes may affect comparisons.
- What statistics are built in?
- Linear mixed models, factorial and repeated-measures ANOVA, logistic GLMM, chi-square, Bayesian tests with Bayes factors, non-parametric tests, equivalence testing (TOST), multiple-comparison corrections, inter-rater reliability, and effect sizes with bootstrap confidence intervals. You can also examine variance decomposition and prompt sensitivity.
- What do I get on the Free plan?
- The current platform-provided AI wizard allowance is temporarily unavailable. The Free plan includes access to the manual study workflow. Check the current plan details for project and feature limits; OpenRouter subject, response-coding judge and optional OpenRouter wizard usage is billed separately.
- Does a model-only study require ethics review?
- Model-inference trials alone do not recruit people. Human annotation or human-derived data can involve separate consent, privacy, or institutional requirements, which vary by context and are outside AIScholar's scope.
- How should I interpret automated coding?
- Automated codes are measurements, not ground truth. Define observable outcomes, inspect disagreements and validate codes for your research question before drawing conclusions.
Read the full FAQ for details on models, costs, data ownership and reproducibility.
Pricing
Start your research for free.
Begin with the free manual study workflow. The current platform-provided AI wizard allowance is temporarily unavailable. Compare plans for more projects and AI-assisted features.
Running trials against subject models uses your own OpenRouter API key. Those OpenRouter usage costs are billed by OpenRouter and are not included in any AIScholar plan. Automated coding/judge calls also use your OpenRouter key and can incur separate charges. Run estimates are not final charges; rates, tokens and retries vary. Review API-key costs.
Free
Run the full research workflow and try AI on the house.
$0
Free forever
- Up to 3 projects
- Platform-provided AI wizard allowance (current count unavailable)
- Full pipeline through Visualization
- Experiment execution & response collection
- Built-in statistical analyses & charts
- Data export (CSV / XLSX)
- AI wizards capped — upgrade for unlimited
- No AI Analyst chat
- No Write-Up assistance
Most popular
Pro
Unlimited AI across every stage — go from idea to manuscript in a day.
$39.99/mo
Checking billing availability…
- Up to 50 projects
- Everything in Free
- Unlimited AI hypothesis, design, prompt & coding-scheme wizards
- AI Analyst — ask your data questions in plain English
- AI Write-Up — citation-backed draft from Intro to Discussion
Enterprise
Everything in Pro, plus compliance-grade R&D documentation for grants & tax credits.
$99.99/mo
Checking billing availability…
- Up to 500 projects
- Everything in Pro
- Reproducibility package export
- R&D documentation module (OECD Frascati-aligned)
- Technological uncertainty tracking
- Effort & expenditure timesheets
- R&D tax-incentive export package
- Documentation supports SR&ED tax credits in Canada
Have a discount code? Enter it at checkout to apply your savings.
Prices in USD. Cancel anytime from the billing portal. Need a custom plan? Contact us.
Your next LLM research question starts here.
Design a study, compare responses and see what the evidence shows, all in one workspace.