LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
By Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica. 2024
1,848 citations recorded by the source corpus.
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model
Subfields: Information Systems, Software, Large Language Models, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Yinlin Deng
- Chun Xia
- Lingming Zhang
- Ling-Ming Zhang
- Soren Dunn
- Joseph E. Gonzalez
- Maosong Sun
- Ben Feuer
In Software
In Large Language Models
In Evaluation & Benchmarks
In Code Generation & Program Synthesis
Citations in the corpus
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, methodology): “Recent works LiveCodeBench [19], R2E [20], as well as the community based live evaluation, Chatbot Arena [9].”
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, methodology): “As shown in the figure, DeepSeek-V2 Chat (RL) demonstrates considerable proficiency in LiveCodeBench, achieving a Pass@1 score that even surpasses some giant models.”
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, neutral): “For evaluation, Models are assessed on LiveCodeBench v5, comprising 880 problems that span from May 2023 to January 2025 (Jain et al., 2024), as well as HumanEval+ and MBPP+ (Liu et al., 2023).”
- Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, neutral): “(3) LiveCodeBench (Jain et al., 2024) is a benchmark for evaluating LLMs’ coding capabilities, consisting of easy, medium, and hard difficulty problems.”
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, positive): “Here, we test whether DAPO, when trained on a math dataset and using only a small fraction of high-entropy tokens in the policy gradient loss, can still surpass vanilla DAPO on out-of-distribution test sets, such as LiveCodeBench (Jain et al., 2024).”
- LiveBench: A Challenging, Contamination-Free LLM Benchmark (2024, methodology): “Other benchmarks Perhaps the most-related benchmark to ours is LiveCodeBench [28], which also regularly releases new questions and makes use of ground-truth judging.”
- Qwen3 Technical Report (2025, neutral): “For example, the flagship model Qwen3-235B-A22B achieves 85.7 on AIME’24 and 81.5 on AIME’25 (AIME, 2025), 70.7 on LiveCodeBench v5 (Jain et al., 2024), 2,056 on CodeForces, and 70.8 on BFCL v3 (Yan et al., 2024).”
- Qwen2.5-Omni Technical Report (2025, positive): “…2024) for general evaluation, GPQA (Rein et al., 2023), GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) for mathematics & science, HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), MultiPL-E (Cassano et al., 2023) and LiveCodeBench 2305-2409 (Jain et al., 2024) for coding.”
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, neutral): “Overall, the retrieved benchmark splits exhibit a long-tail distribution.”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “We compare model performances on BigCodeBench - Complete against existing benchmarks like HumanEval+ [35] and LiveCodeBench [26] used for evaluating the coding capabilities of LLMs.”
- Qwen3-VL Technical Report (2025, positive): “…(Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wang et al., 2025c), (4) Alignment Tasks: IFEval (Zhou et al., 2023), Arena-Hard v2 (Li et al., 2024d) 1 ,…”
- Qwen2 Technical Report (2024, methodology): “The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench.”
- Training Language Models to Self-Correct via Reinforcement Learning (2024, positive): “The most prominent settings include problems where external input tokens from an environment are available, such as agentic tasks (Liu et al., 2023), code repair (Jain et al., 2024), and tool use (Chen et al., 2023).”
- Qwen2.5-Coder Technical Report (2024, background): “5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
- LLMs Get Lost In Multi-Turn Conversation (2025, positive): “The original instructions were sourced from the HumanEval [10] and LiveCodeBench [31] datasets, two popular benchmarks used to evaluate LLM programming aptitude.”
- Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, background): “Furthermore, MT-Bench has been shown to have performance correlated to human preferences of LLM rankings [Zheng et al., 2023, Rafailov et al., 2023, Li et al., 2024].”