Test whether LLM findings survive prompt wording changes
Measure variability across meaning-checked prompt variants, not one paraphrase.
Stage 1 · Hypothesis
The uncomfortable question every LLM result must answer
A measurement based on one hand-written prompt can be sensitive to wording. If a paraphrase would have produced a different number, the result describes this task and prompt as well as the model. This study makes that sensitivity measurable.
| ID | Prediction |
|---|---|
| H1 | A model's measured helpfulness score on a fixed advice task varies significantly across semantically equivalent prompt variants. |
| H2 | The ranking of three subject models is not stable across variants — the "best" model depends on the phrasing used to ask. |
Note the framing: prompt wording is not noise to be minimized but a random effect to be estimated. That makes wording sensitivity measurable rather than leaving it unexamined.
Stage 2 · Design
Variants as a crossed random effect
| Design element | Choice |
|---|---|
| Base task | A fixed personal-finance advice request with a known ground structure (budget, debt, savings priorities) |
| Factor | model (3 OpenRouter models) |
| Random effect | prompt variant (12 variants of the base prompt, 2 per perturbation dimension) |
| Replications | 15 per model × variant cell |
| Expected output | Freeform text (judged for quality downstream) |
| Trial count | 3 models × 12 variants × 15 replications = 540 trials |
The design stage records variants as first-class prompt objects, so every trial knows exactly which wording produced it — the prerequisite for variance decomposition later.
Stage 3 · Prompts
Six perturbation dimensions, generated systematically
The prompt-variant generator produces semantically equivalent rewrites of the base prompt along six dimensions:
| Dimension | Example change |
|---|---|
| Lexical paraphrasing | "I'm struggling to manage my money" → "I'm having a hard time keeping my finances in order" |
| Structural reformatting | One prose paragraph → three bullet points with the same content |
| Verbosity variation | A terse two-sentence ask → a detailed ask with background context |
| Output-format variation | "Reply in plain prose" → "Reply as a numbered plan" |
| Instruction specificity | Open request → request with explicit constraints ("do not recommend new credit products") |
| Persona framing | No role → "You are a certified financial planner" |
I take home $3,400/month. Rent is $1,450, I carry $6,200 in credit-card debt at 24% APR, and I have $800 in savings. What should I prioritize over the next six months?
Every generated variant is shown side-by-side with the base for human review before it enters the study — semantic equivalence is a judgment call the researcher signs off on, not something to delegate blindly.
Stage 4–5 · Execution & Collection
One run, every wording
All 540 trials dispatch in one orchestrated run — same temperature (0.7), same token ceiling — so wording is the only thing that varies within a model. Collection captures response length, latency, and token usage alongside the text; length itself becomes a secondary outcome (do wordier prompts beget wordier answers?).
Stage 6 · Coding
A judged quality score with agreement checks
A judge model scores each response on a five-point helpfulness rubric anchored to concrete criteria (addresses the debt-interest math, sequences actions, avoids generic filler). Because the judge's own wording is a threat here too, the study runs the judge-prompt sensitivity check: rubric paraphrases are applied to a response sample and agreement rates are reported before the main coding pass is trusted.
Stage 7 · Analysis
Variance decomposition and ranking stability
Two analyses answer the two hypotheses directly:
| Variance component | Share of total |
|---|---|
| Model identity | 38% |
| Prompt variant | 22% |
| Model × variant interaction | 9% |
| Residual (within-cell) | 31% |
A decomposition like the illustration would mean nearly a quarter of the measured "helpfulness" is wording — large enough that any single-prompt comparison of these models is unreliable. The ranking-stability analysis then reports, across the 12 variants, how often each model ranks first and the flip rate of the modal ranking; a mixed model with variant as a crossed random effect gives the formal test.
These numbers are illustrative
The example results on this page show what the output of this study design looks like — they are not findings from a real run. Model behavior changes across versions and providers; run the study yourself to get current numbers.
Stage 8 · Write-Up
Reporting sensitivity honestly
The write-up assistant drafts a Results section that reports the variance shares and ranking flip-rate alongside the headline means — the honest way to present any LLM benchmark-style claim. The reproducibility export includes all 12 variant texts verbatim, so readers can audit the perturbations themselves.
Stage ✓ · Honest limits
What this design can and cannot claim
Twelve variants sample the space of reasonable phrasings; they do not exhaust it. The study estimates wording variance for this task family at these sampling parameters — generalizing further requires more task families. And semantic equivalence is asserted by human review, not proven. Pair this design with any substantive study — for instance, re-running the framing-effects study across variants — to turn a one-prompt finding into a robust one.
- Prompt variants
- Variance decomposition
- Ranking stability
More example studies
Psychometric assessment of an LLM
Administering a risk-attitude scale to models with reliability checks, contamination probes, and a validity checklist.Persona × task factorial study
A 3×3 full factorial built with the snippet library and variation grid, analyzed with factorial ANOVA.Coding open-ended responses
LLM-as-a-judge coding with a versioned scheme, multi-judge agreement statistics, and human gold-standard review.
Explore a study of your own.
These walkthroughs illustrate methods, not ready-to-run templates. Plan and review your own design before collecting responses.