Condition summaries
Compare response distributions and report condition counts, missingness, and uncertainty alongside observed differences.
Stage 8 · Analysis & Visualization
Analyze variation in LLM behavior
Compare condition distributions, assess uncertainty and inspect whether wording or model choice changes the finding.
Start with outcomes and denominators
Show the number of valid, coded responses and the denominator for each condition. Report missing and excluded trials transparently; a summary without its denominator can hide missingness or unequal coverage.
Summaries can describe the observed responses. Any inferential method should match the outcome, design, dependence structure, and data quality; no statistical procedure automatically makes a design rigorous or supports generalization beyond the tested conditions.
Choose methods and check assumptions
Inferential methods
Available analyses include methods for categorical, ordinal, and continuous outcomes. Select one only when its assumptions and dependence structure are appropriate to the design.
Model and wording variation
Inspect whether response patterns vary across models or prompt variants. These checks do not turn a limited set of models or prompts into a representative sample of all LLM behavior.
Prompt sensitivity
Compare meaning-checked prompt variants and report whether conclusions depend on particular wording. A paraphrase that changes meaning is a different manipulation, not a robustness check.
Semantic entropy
Examine the spread of meaning-clustered responses by condition and model. Inspect the clusters: a diversity score is not by itself evidence of a model's uncertainty or safety.
Diagnostics
Use diagnostics to question model fit, uncertainty, and the stability of results; investigate warnings rather than treating them as automatic certification.
Coding quality
Interpret coded outcomes alongside human validation, agreement, and sensitivity to the rubric used.
Interpret evidence within its limits
Check coverage and exclusions
Review cell counts, invalid or missing trials, and how each coding decision contributes to the analyzed sample.
Match analysis to the design
Consider outcome type, item/variant structure, dependence, diagnostics, and adequate information before fitting an inferential model. A small or imbalanced study can leave effects uncertain.
Report uncertainty and scope
Present effect estimates and intervals where appropriate, disclose alternative rubric or exclusion choices, and limit conclusions to the prompts, model versions, provider conditions, and single-turn text protocol tested.
The prompt-sensitivity walkthrough shows one related analysis. See also variance decomposition and response coding. When results are ready, the write-up assistant can help prepare a draft for expert review.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.