Multi-judge coding
Run more than one judge or pass and inspect agreement statistics. Agreement is informative about consistency, not proof that the rubric measures the intended construct.
Stage 6 · Response Coding
Code LLM responses with an auditable rubric
Turn each text response into a defined outcome, and check automated judgments against human review.
From raw text to analyzable values
Decide on observable labels before coding. Define edge cases and exclusions in advance; do not force uncertain cases into a binary outcome.
Use structured extraction when the response format allows deterministic parsing. For interpretive outcomes, an LLM-as-a-judge can apply your explicit coding scheme; it is an instrument, not ground truth. Judging can be done with a separate model, but automated labels still require validation.
Compare automated codes and human review
Human review & gold standard
Have knowledgeable reviewers assess a stratified sample, ideally blinded to condition; reconcile disagreements and compare machine codes against reviewed labels.
Versioned coding schemes
Record rubric versions so code changes can be traced and sensitivity to alternative definitions can be examined.
Omission-aware templates
Where relevant, distinguish different observable error types rather than collapsing them into a single label.
Judge-prompt sensitivity check
Explore whether judge labels vary when rubric wording changes; wording checks are a diagnostic and do not establish construct validity.
Coverage tracking
Inspect which responses are coded, pending, or failed so missing coding is visible in the analysis denominators.
Cost, validity, and inference limits
Define the coding scheme
Name the outcome, define labels and ambiguous cases, and document exclusions before examining condition differences.
Select extraction or judge coding
Use deterministic extraction for structured responses where possible. For interpretive labels, select an available judge model and pilot the rubric.
Estimate cost and run
Judge calls use your OpenRouter key and may incur charges separate from subject-model calls. Review current estimates and provider rates; actual cost varies.
Validate and report
Review a sample with human expertise, inspect disagreements and agreement, document exclusions, and test whether plausible rubric choices change the conclusion.
See the judge-coding walkthrough. Related definitions: LLM-as-a-Judge and coding validity. Coded data flows into analysis & visualization.
Study LLM behavior with a guided research workflow.
Design controlled text studies, collect model responses, and explore what changes across conditions. You decide what the evidence means.