Ask bounded questions about safety-relevant model behavior
Examine observable single-turn text behavior related to honesty, sycophancy, uncertainty or refusal. A response pattern is evidence about the tested context, not proof of alignment or deployment safety.
Questions for safety and alignment researchers
Does a model correct a harmless mistaken assertion or agree to please the user (sycophancy)? Does it state uncertainty when evidence is incomplete, rather than sounding certain without grounds? How might observed refusal behavior vary across benign contexts? Truthfulness and honesty concern what a response communicates; calibrated uncertainty concerns whether expressed confidence fits the available evidence. These are distinct questions, and a text label alone cannot validate them.
Start with the AI-safety study idea on authorization: compare how a model judges explicit permission, uncertain permission and unauthorized access when a requester claims the stakes are high.
From observed response to defensible interpretation
Specify the observable behavior of interest and consider alternative explanations before interpreting it. A model may correct one error yet echo another, hedge for unrelated reasons or change its answer under different wording. Inspect responses and disagreements; human review of any coding rubric matters. Refusal behavior on its own does not demonstrate harmlessness, and a wording comparison alone does not establish adversarial robustness.
AIScholar supports controlled conditions, repeated single-turn text calls and response coding, but the researcher remains responsible for construct validity, suitable materials, interpretation and specialist methods or ethics review when appropriate. Avoid treating an automated judge's labels as ground truth.
Conceptual safety & alignment question
Compare a model's judgments about permission, urgency and unauthorized access.Experimental design methods
See how discrete conditions and replication work, not a validated safety protocol.Response coding methods
Define observable codes, inspect disagreements and review automated judgments.Adjacent prompt-sensitivity method
A separate wording-variation walkthrough; it is not an AI-safety study.
Where a specialist evaluation tool fits better
AIScholar may fit a planned, factor-driven study of supported single-turn text responses. Inspect AI supports programmable evaluations involving agents, tools and broader task setups; Promptfoo is designed for application evaluation and iterative red-team testing. These tools overlap in datasets and scoring. See the evaluation-tool comparison.
Scope and costs
Subject models are inference-only, single-turn, text-only calls served through OpenRouter. AIScholar does not evaluate tools, multi-turn conversations, images, private models or model training. Availability and provider behavior can change. Subject-model and automated judge calls use your OpenRouter key and may incur charges; estimates are not final bills. Review cost details and scope limitations.
AIScholar for other researchers
Social & management scholars
Bring experimental rigor to questions about framing, decision-making, and judgment in language models.Psychology researchers
Adapt validated instruments, run psychometric checks, and study machine behavior with familiar methods.Educators & methods instructors
Teach experimental methods with model-response studies while checking institutional requirements for human data and review activities.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.