Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

By Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez and 1 more. 2024

1,398 citations recorded by the source corpus.

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at \url{https://chat.lmsys.org}.

Subfields: Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Yu Meng
2. Fei Huang
3. Micah Goldblum
4. Yann LeCun
5. Jingren Zhou
6. Saining Xie
7. Fuzhao Xue
8. Shusheng Yang

In Evaluation & Benchmarks

1. Yann LeCun
2. Ravid Shwartz-Ziv
3. Fei Huang
4. Yu Meng
5. Jingren Zhou
6. Saining Xie
7. Sinan Tan
8. Shengbang Tong

Citations in the corpus

• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, neutral): “The recent published GPT-4o has achieved remarkable performance boost (10+% improvement) on MATH [19], Chatbot Arena [11].”
• LLMs Get Lost In Multi-Turn Conversation (2025, neutral): “Existing studies implemented such user simulation in different ways: relying on templates [12, 68, 39, 16], using an LLM [63, 46, 7, 48], involving human annotators [21, 7], or real users as part of a study [67, 38, 11].”
• Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “…of novel frameworks for evaluating responses given by agents and LLMs, which reflects a reimagination of how a task should be judged - compared to the prevalently existing evaluation approach by using more capable models to give ratings, or crowd-sourced AI benchmarking from human preference [25].”
• Qwen2 Technical Report (2024, background): “…in evaluations of both fundamental language capabilities and instruction-tuned functionalities Specifically, Qwen2-72B-Instruct, our instruction-tuned variant, scores 9.1 on MT-Bench (Zheng et al., 2023), 48.1 on Arena-Hard (Chiang et al., 2024), and 35.7 on LiveCodeBench (Jain et al., 2024).”
• CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, neutral): “Further implementation details and filtering statistics are provided in Ap-pendices A and D, respectively.”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “To determine if these metrics aligned with real user preferences, we submitted our best-performing model, Gemma-2-9B-it-SimPO, to the Chatbot Arena leaderboard hosted by LMSYS [17].”
• LiveBench: A Challenging, Contamination-Free LLM Benchmark (2024, methodology): “Next, we compare LiveBench to two prominent benchmarks, ChatBot Arena [10] and Arena-Hard [35].”
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “LLM-based “judges” used to evaluate natural language generation on open-ended questions [7, 71, 32] are emerging as a solution to dynamic real-world environments [10, 35] and while showing promise also for computer control agents [39, 16] they remain imperfect evaluators.”
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, positive): “As a comparison, Chatbot Arena [6] serves as a robust benchmark for evaluating chat LLMs.”
• SGLang: Efficient Execution of Structured Language Model Programs (2023, neutral): “SGLang has been deployed in Chatbot Arena [8] to serve open-weight models.”
• LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models (2024, neutral): “Recent benchmarks such as Vibe-Eval [61] and LLaVA-Wilder [32] use real-world data for more authentic testing models abilities in the wild .”
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “This underscores the necessity of developing evaluation protocols like the Chatbot Arena [29] for MLLMs, despite the challenges in collecting large-scale, real-world interaction data.”
• TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “…in performance: no open-recipe models exist in the top 50 of LMSYS’s ChatBotArena (as of November 20th, 2024) have released their post-training data [Chiang et al., 2024], and most open recipes use relatively little data and few rounds of training compared to closed post-training setups, which can…”
• A Large-Language-Model Framework for Automated Humanitarian Situation Reporting (2025, unknown): “This model is selected due to its strong performance across various tasks and its favourable placement in the chatbot arena 50 .”
• Deep Literature Survey Automation with an Iterative Workflow (2025, unknown): “In the LLM evaluation community, similar concerns have led to the development of Chatbot Arena Chiang et al. (2024), which adopts pairwise human preference judgments to overcome the noisiness and inconsistency of absolute ratings.”
• Are Smaller Open-Weight LLMs Closing the Gap to Proprietary Models for Biomedical Question Answering? (2025, unknown): “Today, large open-weight models like Llama 3-405B [9], DeepSeek-V3 [10], and Qwen3-235B-A22B [11] are proving capable of challenging even the best proprietary models, as evidenced by platforms like the LM Arena [12].”
• Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “The LMSYS Chatbot Arena also ranks LLM performance in tasks where they serve as conversational assistants [25].”