Structured Chain-of-Thought Prompting for Code Generation
By Jia Li, Ge Li, Yongming Li, Zhi Jin. ACM Transactions on Software Engineering and Methodology, 2023
358 citations recorded by the source corpus.
Large Language Models (LLMs) have shown impressive abilities in code generation. Chain-of-Thought (CoT) prompting is the state-of-the-art approach to utilizing LLMs. CoT prompting asks LLMs first to generate CoTs (i.e., intermediate natural language reasoning steps) and then output the code. However, the accuracy of CoT prompting still cannot satisfy practical applications. For example, gpt-3.5-turbo with CoT prompting only achieves 53.29% Pass@1 in HumanEval. In this article, we propose Structured CoTs (SCoTs) and present a novel prompting technique for code generation named SCoT prompting. Our motivation is that human developers follow structured programming. Developers use three programming structures (i.e., sequential, branch, and loop) to design and implement structured programs. Thus, we ask LLMs to use three programming structures to generate SCoTs (structured reasoning steps) before outputting the final code. Compared to CoT prompting, SCoT prompting explicitly introduces programming structures and unlocks the structured programming thinking of LLMs. We apply SCoT prompting to two LLMs (i.e., gpt-4-turbo, gpt-3.5-turbo, and DeepSeek Coder-Instruct- \(\{\) 1.3B, 6.7B, 33B \(\}\) ) and evaluate it on three benchmarks (i.e., HumanEval, MBPP, and MBCPP). SCoT prompting outperforms CoT prompting by up to 13.79% in Pass@1. SCoT prompting is robust to examples and achieves substantial improvements. The human evaluation also shows human developers prefer programs from SCoT prompting.
Subfields: Artificial Intelligence, Information Systems, Software
View Paper: https://www.semanticscholar.org/paper/94beb9f249d6d2f1c00d8edfa2db861633aee6f9
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, background): “…with 12 Large Language Models (LLMs), including open-source and closed-source ones, such as AlphaCode [Li et al. , 2022], Incoder [Fried et al. , 2022], CodeGeeX [Zheng et al. , 2023], StarCoder [Li et al. , 2023b], CodeGen-Mono [Nijkamp et al. , 2023b], CodeX [Brown et al. , 2020a], GPT-3.”
- A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications (2024, neutral): “Testing across diverse open-domain question-answering datasets demonstrated notable improvements, including a +7.9 average boost in exact match scores for noisy retrieved documents and a +10.5 enhancement in rejection rates for questions beyond pre-training knowledge.”
- Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “LLMs have been used for various coding-related tasks including code generation and code recommendation [20], [21], [22], [23].”