# When GPT-4 judges chatbot answers, its agreement depends on the matchup

> MT-Bench and Chatbot Arena compare LLM judgments with human preferences, while exposing position, verbosity, and self-enhancement biases.

Canonical: https://aischolar.cc/blog/judging-judges

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# When GPT-4 judges chatbot answers, its agreement depends on the matchup

MT-Bench and Chatbot Arena compare LLM judgments with human preferences, while exposing position, verbosity, and self-enhancement biases.

November 13, 2026

### Two routes to a human preference signal

In [_Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena_](https://arxiv.org/pdf/2306.05685), Zheng and colleagues test whether strong language models can evaluate chatbot responses in open-ended exchanges. The study introduces MT-Bench, a fixed set of multi-turn questions with expert ratings, and analyzes Chatbot Arena, where users compare two anonymously presented chatbots. The released benchmark data includes 3,000 expert votes and 30,000 crowd conversations.

GPT-4's judgments agreed with human votes at rates above 80% in the authors' evaluations. They also found that GPT-4 agreement with human raters rose as the win-rate difference between compared models grew. Single-answer grading tracked pairwise and human preferences well in their tests, giving the authors a second evaluation format to examine.

### Where agreement thins

The aggregate rate sits beside documented position, verbosity, and self-enhancement biases. The agreement gradient across model matchups means that the overall percentage conceals weaker performance on closer comparisons. The work evaluates helpfulness; its authors say safety is largely outside scope, and their score combines dimensions such as accuracy, relevance, and creativity. For MT-Bench and Arena, the relevant unresolved measurement is how reliably judges separate close contenders on each of those dimensions.

### Bibliography

-   Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). _Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena_. arXiv:2306.05685v4 \[cs.CL\]. https://arxiv.org/pdf/2306.05685

Original source: [https://arxiv.org/pdf/2306.05685](https://arxiv.org/pdf/2306.05685)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)