LLM-as-a-subject research, made easier

The easier way to study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. Design conditions, collect repeated responses, code and visualize results, analyze differences and prepare a report in one workspace.

The research pipeline

09 STEPS

Plan

  1. 01Hypotheses
  2. 02Design
  3. 03Prompts

Run

  1. 04Execute
  2. 05Collect
  3. 06Code

Analyze

  1. 07Visualize
  2. 08Analyze
  3. 09Write-up
Why study LLM behavior?

One answer is not the whole story.

As LLMs become part of everyday life, their responses can change with wording, context and model choice. Systematic study shows where patterns hold and where they shift, giving researchers more to work with than a few striking examples.

  • See more than a single answer

    Compare models and conditions over repeated trials to find patterns and variation that one-off prompts miss.

  • Get from question to study faster

    Guided steps and AI assistance help you shape hypotheses, designs, prompts and coding rules without assembling your own toolkit.

  • Measure what matters

    Code observable outcomes with AI and human review, then use built-in statistics and charts to examine the differences.

  • Follow the evidence

    Keep prompts, trial conditions, responses and analysis together so you can see how a finding was produced.

  • Bring your own data

    Upload responses you collected elsewhere and use AIScholar's coding, charts and analysis tools.

  • Take your work with you

    Export your data, charts and an editable report to continue your research or share your work.

How it works

From question to findings, without the setup headache.

AIScholar connects nine stages: hypothesis, design, prompt construction, execution, response collection, coding, visualization, analysis and write-up. You keep control of the method and interpretation.

STAGE 01

Hypotheses

Turn a question about model behavior into predictions you can test.

Learn more about Hypotheses

STAGE 02

Design

Choose conditions and models to compare in a controlled study.

Learn more about Design

STAGE 03

Prompts

Build the prompts for every condition without writing scripts.

Learn more about Prompts

STAGE 04

Execute

Pilot your design, see estimated costs and run repeated text trials.

Learn more about Execute

STAGE 05

Collect

Keep responses and trial details together, ready to investigate.

Learn more about Collect

STAGE 06

Code

Measure responses with AI judges, agreement checks and human review.

Learn more about Code

STAGE 07

Visualize

Explore results through charts, descriptive statistics, and cell coverage.

Learn more about Visualize

STAGE 08

Analyze

Test differences with statistical models and effect sizes.

Learn more about Analyze

STAGE 09

Write-up

Turn your findings into an editable report you can export.

Learn more about Write-up

Explore all features in depth →

What's inside

The tools for the whole study, not just the trial.

Design comparisons, run trials, measure responses, investigate results and prepare a report with tools built to work together.

  • Methodology knowledge base

    Draw on curated research-methods guidance while designing and analyzing your study.

  • Citation traceability

    Keep research references with your project and bring them into your write-up.

  • AI Analyst chat

    Ask questions about your coded data in plain English alongside formal statistical tests.

  • BYOD dataset mode

    Bring responses collected elsewhere into the same coding, visualization and analysis workspace.

Why AIScholar is different

A research workspace, not a stack of disconnected tools.

Different tools solve different parts of the problem. AIScholar brings the research process together in a guided workspace built specifically for studying LLM responses.

vs. LLM eval & observability tools

Eval tools are great for test cases, custom scorers and monitoring applications, including behavioral evaluations.

AIScholar connects a research question to controlled conditions, repeated responses, coding, analysis and a write-up.

vs. generic design-of-experiments software

Design-of-experiments software optimizes experiments; survey platforms collect answers from people.

AIScholar takes your study design straight into LLM text trials, response coding and analysis.

vs. academic research code

A custom notebook gives you full control when you want to build and maintain every step yourself.

AIScholar gives you a guided research workspace without wiring together scripts, spreadsheets and reports.

vs. R&D tax & compliance tooling

Compliance tools help document experimentation for grants or tax reporting.

Run the study and document the work as you go with AIScholar's optional R&D module.

Compare research approaches →

AI assistance

AI help that knows your study.

Start with your own research question. AI assistance can draw on your study to help shape a design, build prompts, suggest coding rules, explore results and draft a write-up, without starting over at every stage.

  • Hypotheses

    Brainstorm and sharpen testable predictions.

  • Design

    Propose factorial conditions and levels.

  • Prompts

    Generate and validate snippet-bound prompt templates.

  • Coding

    Draft coding schemes and judge prompts.

  • Analysis

    Talk through results with the AI Analyst.

  • Write-up

    Draft prose with citations and references.

Bring your own OpenRouter key
Choose the OpenRouter models you want to study and see estimated costs before you run.
AI wizard options
Use platform-provided AI wizard runs within your plan's allowance, or choose OpenRouter with your own key.
Your subject models
Run text trials and AI judge calls through OpenRouter with your key; provider usage is billed separately.
Know before you run
See planned trial counts and subject-model cost estimates before starting a run.
Frequently asked questions

Questions about studying LLMs here.

What exactly can I study on AIScholar?
Study observable outcomes in single-turn text responses from OpenRouter models. Vary prompts, models or supported parameters, repeat trials and code what the models say.
What is out of scope?
Subjects are OpenRouter-served models only: no multi-turn dialogues, tools or agents, multimodal inputs, fine-tuning or model internals. Provider support for settings varies. Model-only trials do not recruit people, but human data or review may require institutional assessment.
Do I need my own API key?
Subject-model trials and automated judge coding incur separate OpenRouter charges. They use your key unless an active project sponsorship authorizes a sponsor's key. Sponsorship changes who pays, not ownership or plan access; model-tier grants are not spending caps. Estimates are not fixed prices. Platform-provided AI is the default. Free includes a limited wizard allowance; Pro and Enterprise include unlimited supported wizard runs. Optional OpenRouter billing for supported wizards and brainstorming is available on all plans through advanced controls, with separate provider charges. Choose personal billing or authorized project sponsorship explicitly; AIScholar does not silently switch payer after a failure. Direct billing does not unlock paid features. AI Analyst, writing agents and live literature search remain platform-managed.
Which models can be research subjects?
Available OpenRouter-served text models can be included as subject conditions. Check availability, exact model string and pricing when you run; provider or version changes may affect comparisons.
What statistics are built in?
Linear mixed models, factorial and repeated-measures ANOVA, logistic GLMM, chi-square, Bayesian tests with Bayes factors, non-parametric tests, equivalence testing (TOST), multiple-comparison corrections, inter-rater reliability, and effect sizes with bootstrap confidence intervals. You can also examine variance decomposition and prompt sensitivity.
What do I get on the Free plan?
The current platform-provided AI wizard allowance is temporarily unavailable. The Free plan includes access to the manual study workflow. Check the current plan details for project and feature limits; OpenRouter subject, response-coding judge and optional OpenRouter wizard usage is billed separately.
Does a model-only study require ethics review?
Model-inference trials alone do not recruit people. Human annotation or human-derived data can involve separate consent, privacy, or institutional requirements, which vary by context and are outside AIScholar's scope.
How should I interpret automated coding?
Automated codes are measurements, not ground truth. Define observable outcomes, inspect disagreements and validate codes for your research question before drawing conclusions.

Read the full FAQ for details on models, costs, data ownership and reproducibility.

Pricing

Start your research for free.

Begin with the free manual study workflow. The current platform-provided AI wizard allowance is temporarily unavailable. Compare plans for more projects and AI-assisted features.

Running trials against subject models uses your own OpenRouter API key. Those OpenRouter usage costs are billed by OpenRouter and are not included in any AIScholar plan. Automated coding/judge calls also use your OpenRouter key and can incur separate charges. Run estimates are not final charges; rates, tokens and retries vary. Review API-key costs.

Free includes the manual study workflow, visualization, built-in statistics and data export. Pro adds AI Analyst and Write-Up assistance. Enterprise adds reproducibility-package export and R&D documentation. Platform subscriptions and OpenRouter usage charges are separate. Subject-model trials and automated judge coding incur separate OpenRouter charges. They use your key unless an active project sponsorship authorizes a sponsor's key. Sponsorship changes who pays, not ownership or plan access; model-tier grants are not spending caps. Platform-provided AI is the default. Free includes a limited wizard allowance; Pro and Enterprise include unlimited supported wizard runs. Optional OpenRouter billing for supported wizards and brainstorming is available on all plans through advanced controls, with separate provider charges. Choose personal billing or authorized project sponsorship explicitly; AIScholar does not silently switch payer after a failure. Direct billing does not unlock paid features. AI Analyst, writing agents and live literature search remain platform-managed.

Get started takes you to sign in. Then choose your plan again. No subscription is purchased until you review and confirm it at checkout.

Free

Run manual studies and try the included AI wizard allowance.

$0

Free forever

  • Up to 3 projects
  • Platform-provided AI wizard allowance (current count unavailable)
  • Manual workflow, visualization & statistical analysis
  • Experiment execution & response collection
  • Built-in statistical analyses & charts
  • Data export (CSV / XLSX)
  • Project information/design export
  • Project data retained for 3 months
  • Included AI wizards capped; optional OpenRouter billing is separate
  • No AI Analyst chat
  • No Write-Up assistance
Pro plan

Pro

Unlimited included wizards, plus AI Analyst and Write-Up.

$39.99/mo

Checking billing availability…

  • Up to 50 projects
  • Everything in Free
  • Project data retained for 12 months
  • Unlimited AI hypothesis, design, prompt & coding-scheme wizards
  • AI Analyst — ask your data questions in plain English
  • AI Write-Up — citation-backed draft from Intro to Discussion

Enterprise

Everything in Pro, plus R&D documentation for grants and tax reporting.

$99.99/mo

Checking billing availability…

  • Up to 500 projects
  • Everything in Pro
  • Project data retained indefinitely
  • Reproducibility package export
  • R&D documentation module (OECD Frascati-aligned)
  • Technological uncertainty tracking
  • Effort & expenditure timesheets
  • R&D tax-incentive export package
  • Documentation supports SR&ED tax credits in Canada

Project data retention

Retention follows the project owner's current plan, measured from the latest completed non-pilot run. Imported data, pilots and design-only edits do not start or extend it. Enterprise and administrator projects are exempt from this expiry. After expiry or deletion, an administrator can recover a project for three calendar months before permanent removal.

Project data retention by plan
PlanData retained for
Free3 months
Pro12 months
EnterpriseExempt from run-based expiry

Have a discount code? Enter it at checkout to apply your savings.

Prices in USD. Applicable taxes are added at checkout. Cancel anytime from the billing portal. Need a custom plan? Contact us.

Your next LLM research question starts here.

Design a study, compare responses and see what the evidence shows, all in one workspace.