# Chatbot Arena ranks models through anonymous head-to-head votes

> The live evaluation platform collected more than 240,000 votes and compared crowd preferences with expert ratings.

Canonical: https://aischolar.cc/blog/chatbot-arena-human-preference

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# Chatbot Arena ranks models through anonymous head-to-head votes

The live evaluation platform collected more than 240,000 votes and compared crowd preferences with expert ratings.

October 2, 2026

### A live prompt becomes a pairwise vote

Chatbot Arena asks a user to submit a prompt, then presents two anonymous chatbot replies for comparison. In [_Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference_](https://arxiv.org/pdf/2403.04132), Chiang and colleagues analyze more than 240,000 votes collected during the platform's first months. Their ranking method uses those preferences to estimate model performance; they also compare crowd votes with expert ratings and report good agreement.

### The voters shape the measure

Arena's questions come from participating users, rather than a fixed evaluation set. Conversations in the analyzed data span more than 100 languages, with 77% in English. The vote stream therefore describes choices made by this platform's users and the prompts they supplied. It does not answer whether a preferred response is factually correct or safe; each vote records a participant's choice for a particular exchange. That 77% share is a useful boundary to retain when interpreting Arena's multilingual coverage.

### Bibliography

-   Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., & Stoica, I. (2024). _Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference_. arXiv:2403.04132v1 \[cs.AI\]. https://arxiv.org/pdf/2403.04132

Original source: [https://arxiv.org/pdf/2403.04132](https://arxiv.org/pdf/2403.04132)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
