Explaining Competitive-Level Programming Solutions using LLMs

By Jierui Li, Szymon Tworkowski, Yingying Wu, Raymond J. Mooney. 2023

19 citations recorded by the source corpus.

In this paper, we approach competitive-level programming problem-solving as a composite task of reasoning and code generation. We propose a novel method to automatically annotate natural language explanations to \textit{} pairs. We show that despite poor performance in solving competitive-level programming problems, state-of-the-art LLMs exhibit a strong capacity in describing and explaining solutions. Our explanation generation methodology can generate a structured solution explanation for the problem containing descriptions and analysis. To evaluate the quality of the annotated explanations, we examine their effectiveness in two aspects: 1) satisfying the human programming expert who authored the oracle solution, and 2) aiding LLMs in solving problems more effectively. The experimental results on the CodeContests dataset demonstrate that while LLM GPT3.5's and GPT-4's abilities in describing the solution are comparable, GPT-4 shows a better understanding of the key idea behind the solution.

Subfields: Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Ion Stoica
2. Naman Jain
3. Tianjun Zhang
4. Armando Solar-Lezama
5. Joseph E. Gonzalez

In Code Generation & Program Synthesis

1. Armando Solar-Lezama
2. Koushik Sen
3. Joseph E. Gonzalez
4. Rishabh Singh

Citations in the corpus

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…2(Gemini Team et al., 2023), ALGO (Zhang et al., 2023d), Parsel (Zelikman et al., 2022), code cleaning (Jain et al., 2023), code explainations (Li et al., 2023a), analogical reasoning (Yasunaga et al., 2023), and AlphaCodium (Ridnik et al., 2024) have been pushing the boundaries of what is…”
• LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, methodology): “Li et al. (2023a) also study disentangling the planning and code generation capabilities for closed source LLM S , similar to our experiments on open models.”