CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

By Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, Sida Wang. 2024

322 citations recorded by the source corpus.

We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output pair, leading to two natural tasks: input prediction and output prediction. First, we propose a generic recipe for generating our execution benchmark which can be used to create future variation of the benchmark. Second, we evaluate twenty code models on our benchmark and discover that many recent high-scoring models on HumanEval do not show the same improvements on our benchmark. Third, we show that simple CoT and fine-tuning schemes can improve performance on our benchmark but remain far from solving it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively. In contrast, Code Llama 34B achieves a pass@1 of 50% and 46% on input and output prediction, highlighting the gap between open and closed source models. As no model is close to acing CRUXEval, we provide examples of consistent GPT-4 failures on simple programs as a lens into its code reasoning capabilities and areas for improvement.

Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Information Systems

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “We draw inspiration from the benchmark creation procedure used in Gu et al. (2024).”
R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, unknown): “…reasoning models like Deepseek-R1 [8], chain-of-thought prompting [28], and Cumulative Reasoning [39] have achieved remarkable progress in solving complex problems including coding [3, 7], mathematics [5, 10], This ICCV paper is the Open Access version, provided by the Computer Vision Foundation.”
Qwen2.5-Coder Technical Report (2024, methodology): “For both CRUXEval-I and CRUXEval-O, we used a chain-of-thought (CoT) approach, requiring the LLM to output steps sequentially during simulated execution.”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, methodology): “CodeMind is more closely related to the very recent studies focusing on code reasoning (La Malfa et al., 2024; Gu et al., 2024; Zhang et al., 2024).”
Qwen3 Technical Report (2025, unknown): “…(Average of HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), Humaneval+, MBPP+) (Liu et al., 2023a), MultiPL-E (Cassano et al., 2023) (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript), MBPP-3shot (Austin et al., 2021), CRUX-O of CRUXEval (1-shot) (Gu et al., 2024).”
AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming (2025, unknown): “Shi et al. [2024] showed that including in-context exemplars of solved problems can enhance LLMs’ abilities to solve competitive programming problems.”
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis (2025, unknown): “Beyond generation, tasks like I/O prediction (Gu et al., 2024; Liu et al., 2024a), execution prediction (Liu et al., 2023a; La Malfa et al., 2024; Ni et al., 2024a; Ding et al., 2024), bug localization (Suresh et al., 2025a), and program equivalence (Wei et al., 2025a) have also been studied.”