Any OpenRouter model
Choose subject models currently available through OpenRouter; availability and model identifiers can change.
Stages 4–5 · Execution & Collection
Run replicated text-only LLM trials via OpenRouter
Pilot conditions, inspect estimated subject-model cost, then dispatch the planned cells and retain trial metadata.
Plan trials from cells, models, and replications
A trial is one model call for a resolved prompt and parameter set. Plan volume by multiplying the number of eligible cells by subject models and repetitions. For example, 12 items × 2 request formats × 2 models × 2 wording variants × 10 replications would mean 960 planned calls. This arithmetic is illustrative; confirm the resolved design and model availability before dispatch.
Repeated trials characterize responses under the specified conditions, but they do not make repeated versions of one item equivalent to independent topics. The analysis and generalization remain bounded by the prompts, models, and protocol you chose.
Pilot, review, then dispatch
Inspect the planned grid
Review resolved conditions, prompts, subject models, parameters, trial count, and the estimated subject-call cost before launch.
Pilot selected conditions
A pilot can expose prompt-resolution, parsing, or availability issues. A single response per condition is a setup check, not a substitute for planned replications.
Run and monitor
Dispatch supported trials concurrently. Transient failures may be retried; inspect retry and invalid-response records rather than silently treating them as valid observations.
Retain trial records
Review each response with its prompt, model string, parameters, provider seed support where available, timing, tokens, cost, and validity information.
Costs, keys, and provider variation
Sampling parameters
Set supported sampling parameters such as temperature, top-p, max tokens, and seed. A requested seed may not be honored by every provider.
Concurrency & retries
Parallel dispatch and retry handling for transient API failures; inspect incomplete and retried trials in the run records.
Pilot mode
Use a pilot to check setup before a full run. It still makes paid provider calls and is not a precise estimate of outcome rates.
Cost estimation
Review an estimate based on the planned calls and token budgets. Actual charges vary with usage and provider rates; estimates are not a price guarantee.
Bring your own key
Subject-model calls use your OpenRouter key and incur provider charges. Automated judge calls also use your key and can incur separate charges; plan limits for platform features may apply.
Interpret provider metadata carefully
Explore the prompt-sensitivity walkthrough and review the variable-cost guidance. Collected responses flow into response coding; see also recorded study provenance.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.