Questions about studying LLM behavior with AIScholar

Questions and answers

What is AIScholar?

AIScholar is a guided research workflow for designing and running controlled studies of LLM behavior. Researchers treat OpenRouter-served models as subjects, vary text prompts and settings, replicate single-turn trials, code text responses, analyze results, and document their method. Human researchers remain responsible for the question, valid measurement, review and interpretation.See all features

What kinds of studies can I run?

Supported studies use single-turn, text-only responses from inference-only models available through OpenRouter. You can vary discrete prompt or model settings, replicate trials and measure outcomes derivable from each text response, such as labels, numbers, refusal coding or response length. Results describe the chosen prompts, models, settings and measurements; they do not establish broad claims about model behavior or real-world safety.Explore an illustrative example

Which models can be research subjects?

Subject models must be available through OpenRouter. Provider access, exact model versions and behavior can change; AIScholar runs inference calls and does not fine-tune models or provide access to their internals.

What is out of scope?

AIScholar supports single-turn, text-only inference calls to models available through OpenRouter. It does not support multi-turn dialogue, tool use or agentic loops, image, audio or video inputs, fine-tuning, private/local subject models, or access to model internals. Outcomes must be derived from a single text response. These product limits do not determine whether a research protocol involving people or human-derived material needs review.

Do I need my own API key, and how do costs work?

Subject-model trials use your own OpenRouter API key (BYOK) and are billed by OpenRouter at its current provider rates. LLM-as-a-Judge coding calls also use your OpenRouter key and can incur separate provider charges; they are distinct from subject-model trial costs. Platform-provided AI wizard runs follow your plan's allowance; choosing OpenRouter for wizard calls instead uses your key and incurs separate provider charges. Estimates can help you plan, but actual spending varies with provider rates, selected models, prompt and response length, retries and the calls you run. Pilot mode samples one response per selected condition-and-model cell; it is a design check, not a reliable estimate of an outcome rate.What BYOK meansHow execution worksHow judge coding works

How much does a typical study cost to run?

There is no fixed typical cost: spending varies with current provider rates, subject and judge models, prompt and response lengths, the number of condition × model × replication calls, and retries. Review the estimate before running, then check your OpenRouter account for actual charges; provider billing and spending limits apply. Estimates are not guarantees.

What is the difference between the Free, Pro, and Enterprise plans?

Plan features, limits and prices can change. See the current pricing and plan details before choosing a plan; plan limits apply to AIScholar features and are separate from OpenRouter charges for subject-model and judge calls.View current plans and pricing

How are open-ended responses turned into data I can trust?

Structured outcomes can be extracted directly (labels, numbers, JSON or regex); interpretive outcomes can use LLM-as-a-Judge coding against a written rubric. Human review, agreement checks and validation on an appropriate sample help assess the codes; an automated judge is not ground truth, and researchers remain responsible for the codebook and interpretation.Response coding featuresWhat LLM-as-a-Judge means

Does a model-only study require ethics review?

Model-inference trials alone do not recruit people; that does not determine the review status of the full protocol. Human recruitment or annotation, interaction with people, and identifiable or other human-derived data may raise review, consent, privacy or data-handling questions. Institutional, sponsor and local rules may also apply. Before starting, ask the responsible research-ethics or compliance office (such as an IRB or research ethics committee where applicable) for a protocol-specific determination. Requirements vary by location; AIScholar cannot determine review status.Review study scope

Is AIScholar suitable for teaching research methods?

AIScholar can support course exercises using model-inference trials without recruiting people for those calls. A course setting does not automatically waive review or policy requirements. Human recruitment or annotation and human-derived data may raise consent, privacy or review questions; institutional, sponsor and local rules vary. Check with the responsible research-ethics or compliance office before collecting such data. AIScholar does not determine review status.AIScholar for educators

Who owns my research data?

You do. Your designs, prompts, responses, coded data, and analyses belong to you, and you can export raw and coded data as CSV or XLSX at any time. Enterprise subscribers can also export a reproducibility package per study. Sensitive credentials such as your OpenRouter API key are encrypted at rest with AES-256-GCM.Privacy Policy

How does AIScholar support reproducibility?

AIScholar versions prompt templates and coding schemes, and trial records can include resolved prompts, model identifiers, parameters, seed status, token usage and cost. Studies can export a Method Specification Prompt; Enterprise subscribers can also export a reproducibility package to help others inspect what was run. Provider and model changes can still affect replication.Reproducibility featuresWhat an MSP is

What statistics are built in?

Available analyses include linear mixed models, factorial and repeated-measures ANOVA, logistic GLMM, chi-square, Bayesian tests, non-parametric tests, equivalence testing, multiple-comparison corrections, inter-rater reliability and effect sizes with confidence intervals, along with LLM-specific analyses such as variance decomposition, prompt sensitivity and semantic entropy. Choose methods appropriate to your design and inspect their assumptions.Analysis & visualization features

Can I analyze data I collected outside AIScholar?

Yes. The bring-your-own-data workflow imports LLM responses collected elsewhere: you upload the dataset, map columns to conditions, code responses and run analyses. Imported data do not retroactively acquire randomized assignment or complete provenance, so interpret results in light of the original collection method.Walkthrough: analyzing an existing dataset

Explore an illustrative prompt-sensitivity example, or review execution costs and scope. Looking for definitions instead? Try the methodology glossary, or see how AIScholar compares to neighbouring tools.