# An Easier Way to Study LLM Behavior · AIScholar

> Study LLM behavior with controlled, replicated single-turn text trials. Design prompts, collect responses, code, visualize, analyze and report findings.

Canonical: https://aischolar.cc/

LLM-as-a-subject research, made easier

# The easier way to study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. Design conditions, collect repeated responses, code and visualize results, analyze differences and prepare a report in one workspace.

[Start a study](https://aischolar.cc/sign-in)[See an example study](https://aischolar.cc/study-ideas/money-manager-bonus)

## The research pipeline

09 STEPS

### Plan

1.  01Hypotheses
2.  02Design
3.  03Prompts

### Run

1.  04Execute
2.  05Collect
3.  06Code

### Analyze

1.  07Visualize
2.  08Analyze
3.  09Write-up

Why study LLM behavior?

## One answer is not the whole story.

As LLMs become part of everyday life, their responses can change with wording, context and model choice. Systematic study shows where patterns hold and where they shift, giving researchers more to work with than a few striking examples.

-   ### See more than a single answer

    Compare models and conditions over repeated trials to find patterns and variation that one-off prompts miss.

-   ### Get from question to study faster

    Guided steps and AI assistance help you shape hypotheses, designs, prompts and coding rules without assembling your own toolkit.

-   ### Measure what matters

    Code observable outcomes with AI and human review, then use built-in statistics and charts to examine the differences.

-   ### Follow the evidence

    Keep prompts, trial conditions, responses and analysis together so you can see how a finding was produced.

-   ### Bring your own data

    Upload responses you collected elsewhere and use AIScholar's coding, charts and analysis tools.

-   ### Take your work with you

    Export your data, charts and an editable report to continue your research or share your work.

How it works

## From question to findings, without the setup headache.

AIScholar connects nine stages: hypothesis, design, prompt construction, execution, response collection, coding, visualization, analysis and write-up. You keep control of the method and interpretation.

STAGE 01

### Hypotheses

Turn a question about model behavior into predictions you can test.

[Learn more about Hypotheses](https://aischolar.cc/features/experimental-design)

STAGE 02

### Design

Choose conditions and models to compare in a controlled study.

[Learn more about Design](https://aischolar.cc/features/experimental-design)

STAGE 03

### Prompts

Build the prompts for every condition without writing scripts.

[Learn more about Prompts](https://aischolar.cc/features/prompt-construction)

STAGE 04

### Execute

Pilot your design, see estimated costs and run repeated text trials.

[Learn more about Execute](https://aischolar.cc/features/experiment-execution)

STAGE 05

### Collect

Keep responses and trial details together, ready to investigate.

[Learn more about Collect](https://aischolar.cc/features/experiment-execution)

STAGE 06

### Code

Measure responses with AI judges, agreement checks and human review.

[Learn more about Code](https://aischolar.cc/features/response-coding)

STAGE 07

### Visualize

Explore results through charts, descriptive statistics, and cell coverage.

[Learn more about Visualize](https://aischolar.cc/features/analysis)

STAGE 08

### Analyze

Test differences with statistical models and effect sizes.

[Learn more about Analyze](https://aischolar.cc/features/analysis)

STAGE 09

### Write-up

Turn your findings into an editable report you can export.

[Learn more about Write-up](https://aischolar.cc/features/write-up)

[Explore all features in depth →](https://aischolar.cc/features)

What's inside

## The tools for the whole study, not just the trial.

Design comparisons, run trials, measure responses, investigate results and prepare a report with tools built to work together.

-   [

    ### Hypotheses & experimental design

    Choose factorial, fractional, matched-pair or custom conditions. Use guided tools and AI assistance to turn a question into a study design.

    Learn more](https://aischolar.cc/features/experimental-design)
-   [

    ### Prompt construction

    Build prompts from reusable snippets, edit variations in a grid and preview what each condition will send before running trials.

    Learn more](https://aischolar.cc/features/prompt-construction)
-   [

    ### Experiment execution

    Pilot your conditions, review estimated costs and run repeated text trials across available OpenRouter models.

    Learn more](https://aischolar.cc/features/experiment-execution)
-   [

    ### Response coding

    Define what you want to measure, code responses with LLM judges, compare agreement and review or correct codes yourself.

    Learn more](https://aischolar.cc/features/response-coding)
-   [

    ### Analysis & visualization

    See patterns in charts and use mixed models, ANOVA, GLMM, Bayesian tests, effect sizes and LLM-specific checks to investigate them.

    Learn more](https://aischolar.cc/features/analysis)
-   [

    ### Write-up assistance

    Draft sections from your study data, work with citations and edit the manuscript before exporting to Word, Markdown or LaTeX.

    Learn more](https://aischolar.cc/features/write-up)
-   [

    ### Reproducibility

    Keep the prompts, conditions, trial settings, responses and coding decisions behind your results together in an inspectable research record.

    Learn more](https://aischolar.cc/features/reproducibility)
-   [

    ### R&D documentation

    OECD Frascati-aligned uncertainty statements, a contemporaneous activity log, effort timesheets, and an export package built for R&D tax-incentive and grant reporting.

    Learn more](https://aischolar.cc/features/rd-documentation)

-   ### Methodology knowledge base

    Draw on curated research-methods guidance while designing and analyzing your study.

-   ### Citation traceability

    Keep research references with your project and bring them into your write-up.

-   ### AI Analyst chat

    Ask questions about your coded data in plain English alongside formal statistical tests.

-   ### BYOD dataset mode

    Bring responses collected elsewhere into the same coding, visualization and analysis workspace.

Why AIScholar is different

## A research workspace, not a stack of disconnected tools.

Different tools solve different parts of the problem. AIScholar brings the research process together in a guided workspace built specifically for studying LLM responses.

### vs. LLM eval & observability tools

Eval tools are great for test cases, custom scorers and monitoring applications, including behavioral evaluations.

AIScholar connects a research question to controlled conditions, repeated responses, coding, analysis and a write-up.

### vs. generic design-of-experiments software

Design-of-experiments software optimizes experiments; survey platforms collect answers from people.

AIScholar takes your study design straight into LLM text trials, response coding and analysis.

### vs. academic research code

A custom notebook gives you full control when you want to build and maintain every step yourself.

AIScholar gives you a guided research workspace without wiring together scripts, spreadsheets and reports.

### vs. R&D tax & compliance tooling

Compliance tools help document experimentation for grants or tax reporting.

Run the study and document the work as you go with AIScholar's optional R&D module.

[Compare research approaches →](https://aischolar.cc/compare)

AI assistance

## AI help that knows your study.

Start with your own research question. AI assistance can draw on your study to help shape a design, build prompts, suggest coding rules, explore results and draft a write-up, without starting over at every stage.

-   ### Hypotheses

    Brainstorm and sharpen testable predictions.

-   ### Design

    Propose factorial conditions and levels.

-   ### Prompts

    Generate and validate snippet-bound prompt templates.

-   ### Coding

    Draft coding schemes and judge prompts.

-   ### Analysis

    Talk through results with the AI Analyst.

-   ### Write-up

    Draft prose with citations and references.

Bring your own OpenRouter key

Choose the OpenRouter models you want to study and see estimated costs before you run.

AI wizard options

Use platform-provided AI wizard runs within your plan's allowance, or choose OpenRouter with your own key.

Your subject models

Run text trials and AI judge calls through OpenRouter with your key; provider usage is billed separately.

Know before you run

See planned trial counts and subject-model cost estimates before starting a run.

Example studies

## See what you could investigate.

Explore literature-grounded Study Ideas about LLM behavior. Each post includes a “Conduct this Study” button that sets up its hypotheses, research design and prompts in your account.

-   [

    ### Financial incentives & LLM advice

    Test how an upside-only bonus changes the financial advice LLMs give across three supplied cases.

    ](https://aischolar.cc/study-ideas/money-manager-bonus)
-   [

    ### Party labels & LLM policy support

    Compare LLM support for the same policy when its party attribution changes.

    ](https://aischolar.cc/study-ideas/politics-party-label)
-   [

    ### Student labels & LLM evaluations

    Investigate how a gifted label changes LLM evaluations of identical student work.

    ](https://aischolar.cc/study-ideas/education-student-label)
-   [

    ### Public scrutiny & LLM decisions

    Examine how public scrutiny changes LLM advice about a profitable organizational policy.

    ](https://aischolar.cc/study-ideas/organization-public-scrutiny)

[Browse all example studies](https://aischolar.cc/study-ideas)

Who it's for

## For everyone studying how LLMs behave.

-   [

    ### Social & management scholars

    Run controlled studies of framing, decision-making and social behavior in language models.

    ](https://aischolar.cc/for/social-science-researchers)
-   [

    ### LLM behavior researchers

    Compare observable patterns in honesty, uncertainty, refusal and other safety-relevant responses.

    ](https://aischolar.cc/for/ai-safety-researchers)
-   [

    ### Psychology researchers

    Adapt established measures to model responses and investigate reliability and consistency.

    ](https://aischolar.cc/for/psychology-researchers)
-   [

    ### Students & educators

    Teach and learn the full research process through hands-on LLM studies.

    ](https://aischolar.cc/for/educators)

Frequently asked questions

## Questions about studying LLMs here.

### What exactly can I study on AIScholar?

Study observable outcomes in single-turn text responses from OpenRouter models. Vary prompts, models or supported parameters, repeat trials and code what the models say.

### What is out of scope?

Subjects are OpenRouter-served models only: no multi-turn dialogues, tools or agents, multimodal inputs, fine-tuning or model internals. Provider support for settings varies. Model-only trials do not recruit people, but human data or review may require institutional assessment.

### Do I need my own API key?

Subject-model trials and automated judge coding incur separate OpenRouter charges. They use your key unless an active project sponsorship authorizes a sponsor's key. Sponsorship changes who pays, not ownership or plan access; model-tier grants are not spending caps. Estimates are not fixed prices. Platform-provided AI is the default. Free includes a limited wizard allowance; Pro and Enterprise include unlimited supported wizard runs. Optional OpenRouter billing for supported wizards and brainstorming is available on all plans through advanced controls, with separate provider charges. Choose personal billing or authorized project sponsorship explicitly; AIScholar does not silently switch payer after a failure. Direct billing does not unlock paid features. AI Analyst, writing agents and live literature search remain platform-managed.

### Which models can be research subjects?

Available OpenRouter-served text models can be included as subject conditions. Check availability, exact model string and pricing when you run; provider or version changes may affect comparisons.

### What statistics are built in?

Linear mixed models, factorial and repeated-measures ANOVA, logistic GLMM, chi-square, Bayesian tests with Bayes factors, non-parametric tests, equivalence testing (TOST), multiple-comparison corrections, inter-rater reliability, and effect sizes with bootstrap confidence intervals. You can also examine variance decomposition and prompt sensitivity.

### What do I get on the Free plan?

The current platform-provided AI wizard allowance is temporarily unavailable. The Free plan includes access to the manual study workflow. Check the current plan details for project and feature limits; OpenRouter subject, response-coding judge and optional OpenRouter wizard usage is billed separately.

### Does a model-only study require ethics review?

Model-inference trials alone do not recruit people. Human annotation or human-derived data can involve separate consent, privacy, or institutional requirements, which vary by context and are outside AIScholar's scope.

### How should I interpret automated coding?

Automated codes are measurements, not ground truth. Define observable outcomes, inspect disagreements and validate codes for your research question before drawing conclusions.

[Read the full FAQ](https://aischolar.cc/faq) for details on models, costs, data ownership and reproducibility.

Pricing

## Start your research for free.

Begin with the free manual study workflow. The current platform-provided AI wizard allowance is temporarily unavailable. Compare plans for more projects and AI-assisted features.

Running trials against subject models uses your own OpenRouter API key. Those OpenRouter usage costs are billed by OpenRouter and are not included in any AIScholar plan. Automated coding/judge calls also use your OpenRouter key and can incur separate charges. Run estimates are not final charges; rates, tokens and retries vary. [Review API-key costs](https://aischolar.cc/faq#api-key-costs).

Free includes the manual study workflow, visualization, built-in statistics and data export. Pro adds AI Analyst and Write-Up assistance. Enterprise adds reproducibility-package export and R&D documentation. Platform subscriptions and OpenRouter usage charges are separate. Subject-model trials and automated judge coding incur separate OpenRouter charges. They use your key unless an active project sponsorship authorizes a sponsor's key. Sponsorship changes who pays, not ownership or plan access; model-tier grants are not spending caps. Platform-provided AI is the default. Free includes a limited wizard allowance; Pro and Enterprise include unlimited supported wizard runs. Optional OpenRouter billing for supported wizards and brainstorming is available on all plans through advanced controls, with separate provider charges. Choose personal billing or authorized project sponsorship explicitly; AIScholar does not silently switch payer after a failure. Direct billing does not unlock paid features. AI Analyst, writing agents and live literature search remain platform-managed.

Get started takes you to sign in. Then choose your plan again. No subscription is purchased until you review and confirm it at checkout.

### Free

Run manual studies and try the included AI wizard allowance.

$0

Free forever

[Get started](https://aischolar.cc/sign-in)

-   Up to 3 projects
-   Platform-provided AI wizard allowance (current count unavailable)
-   Manual workflow, visualization & statistical analysis
-   Experiment execution & response collection
-   Built-in statistical analyses & charts
-   Data export (CSV / XLSX)
-   Project information/design export
-   Project data retained for 3 months
-   Included AI wizards capped; optional OpenRouter billing is separate
-   No AI Analyst chat
-   No Write-Up assistance

Pro plan

### Pro

Unlimited included wizards, plus AI Analyst and Write-Up.

$39.99/mo

Checking billing availability…

-   Up to 50 projects
-   Everything in Free
-   Project data retained for 12 months
-   Unlimited AI hypothesis, design, prompt & coding-scheme wizards
-   AI Analyst — ask your data questions in plain English
-   AI Write-Up — citation-backed draft from Intro to Discussion

### Enterprise

Everything in Pro, plus R&D documentation for grants and tax reporting.

$99.99/mo

Checking billing availability…

-   Up to 500 projects
-   Everything in Pro
-   Project data retained indefinitely
-   Reproducibility package export
-   R&D documentation module (OECD Frascati-aligned)
-   Technological uncertainty tracking
-   Effort & expenditure timesheets
-   R&D tax-incentive export package
-   Documentation supports SR&ED tax credits in Canada

### Project data retention

Retention follows the project owner's current plan, measured from the latest completed non-pilot run. Imported data, pilots and design-only edits do not start or extend it. Enterprise and administrator projects are exempt from this expiry. After expiry or deletion, an administrator can recover a project for three calendar months before permanent removal.

| Plan | Data retained for |
| --- | --- |
| Free | 3 months |
| Pro | 12 months |
| Enterprise | Exempt from run-based expiry |

Have a discount code? Enter it at checkout to apply your savings.

Prices in USD. Applicable taxes are added at checkout. Cancel anytime from the billing portal. Need a custom plan? [Contact us](mailto:info@digitvibe.com).

## Your next LLM research question starts here.

Design a study, compare responses and see what the evidence shows, all in one workspace.

[Start a study](https://aischolar.cc/sign-in)
