Semantic uncertainty

Because a language model's output is sampled from a probability distribution, asking the same question repeatedly yields varying answers. Raw text differences overstate this variation — "Yes, that's correct" and "That is right" are different strings but the same answer. Semantic uncertainty fixes this by first grouping responses into meaning clusters, then measuring the spread of the distribution over clusters. Entropy is zero when every response means the same thing and rises as answers scatter across distinct meanings.

High semantic uncertainty in an experimental condition is diagnostic. It can indicate that the prompt is ambiguous, that the model genuinely has no stable disposition on the question, or that temperature is set too high for the measurement you want. Either way, a condition where the model gives ten semantically different answers should not be summarized by a single "typical" response — the diversity itself is the finding.

Consistency is therefore a first-class outcome variable, not just noise. Studies can compare semantic entropy across conditions, models, or parameter settings — for example, testing whether higher temperature increases the semantic diversity of answers to open-ended questions, or which of several models answers a class of questions most consistently.

• Variance decomposition

  Partitioning outcome variance into condition, model, prompt-variant, and residual components.

• Prompt sensitivity

  Measuring how results change under semantically-equivalent rewordings of a prompt.

• LLM-as-a-Judge

  Using a judge model to code free-text responses against a rubric, with human calibration.

Browse all 11 glossary terms

AIScholar offers semantic-entropy analysis by condition and model using clustered responses. Clustering can be AI-assisted or use a text-similarity fallback; check the groups before interpreting the score.

• Analysis & visualization features