OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
By Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue. 2024
279 citations recorded by the source corpus.
The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.
Subfields: Artificial Intelligence, Software, LLM Agents & Tool Use, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Software
- Chun Xia
- Lingming Zhang
- Yinlin Deng
- Ling-Ming Zhang
- Soren Dunn
- Armando Solar-Lezama
- Koushik Sen
- Naman Jain
In Code Generation & Program Synthesis
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…2023; Wei et al., 2023b; Ridnik et al., 2024; Lozhkov et al., 2024) and their applications across various domains and tasks such as program repair (Zheng et al., 2024; Olausson et al., 2023), optimization (Madaan et al., 2023a), test generation (Steenhoek et al., 2023), documentation generation…”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “The outcomes of these experiments are depicted in Figure 7, with additional results presented in Appendix K. Figure 7a illustrates the quantization error reduction ratio across various ranks.”
- Executable Code Actions Elicit Better LLM Agents (2024, methodology): “The following is an example tool definition for web-browsing.”
- Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “A GENTLESS then inputs the processed repository structure along with the original issue description to an LLM, and requests it to identify a list of the top N suspicious files that need further inspection or modification to resolve the issue.”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…(en) [44], OpenAI-Summarize-TLDR (en) [21], Know-Saraswati-CoT (en) [114], General QA FLAN (en) [258], FLANv2 (en & zh) [41] Code-Feedback (en) [311], Glaive-Code-Assistant (en) [73], XCoder-80K (en) [255], LeetCode (en & zh), Code Evol-Instruct-Code (en) [173], InternLM2-Code (en & zh) [19]…”
- MAmmoTH2: Scaling Instructions from the Web (2024, unknown): “The open-source datasets we incorporate are OpenHermes 2.5 [Teknium, 2023], Code-Feedback [Zheng et al., 2024c] and our Math-Plus, which is an augmented version of MetaMathQA (395K) [Yu et al., 2023] and Orca-Math (200K) [Mitra et al., 2024].”
- EmbeddingRWKV: State-Centric Retrieval with Reusable States (2026, unknown): “…(Devlin et al., 2019; Hou et al., 2024a) to large language model backbones, with recent systems such as NV-Emb (Lee et al., 2024), E5-Mistral (Wang et al., 2024c), and GTE (Zhang et al., 2024) achieving strong performance on benchmarks like MTEB (Muennighoff et al., 2023a; Xiao et al., 2023).”
- jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval (2025, unknown): “Training triplets are derived from a variety of sources, including CodeSearchNet, CodeFeedback [Zheng et al., 2024], APPS [Hendrycks et al., 2021], and the CornStack dataset [Suresh et al., 2025].”