LLM-as-a-Judge coding and its limits
LLM-as-a-Judge is a coding method in which a judge language model assigns structured labels or scores to another model's text response using a written rubric. Its outputs are judgments to validate, not ground truth.
What "LLM-as-a-Judge" means
The judge receives a response and explicit coding instructions, then returns a structured label or score. Researchers define the codebook, inspect disagreements and validate a sample against human coding where appropriate.
In a response-coding study, a judge can apply a researcher-defined rubric to produce labels for later inspection.
Automated codes are rubric-dependent judgments; they require researcher validation and do not, by themselves, establish the underlying construct.
Related terms
Psychometric validity for LLMs
Re-establishing reliability and validity when human instruments are administered to models.Semantic uncertainty
Shannon entropy over meaning-clustered responses — response consistency as a measurable outcome.Prompt sensitivity
Measuring how results change under semantically-equivalent rewordings of a prompt.
How AIScholar uses it
AIScholar supports rubric-based judge coding with OpenRouter models, human review and agreement analysis. Judge calls use the researcher's OpenRouter key and may incur charges separate from subject-model calls.
See the method in action.
Explore an illustrative prompt-sensitivity study, then review execution costs and scope before planning your own.