Automatic Chain of Thought Prompting in Large Language Models
By Zhuosheng Zhang, Aston Zhang, Mu Li, Alexander J. Smola. 2022
1,007 citations recorded by the source corpus.
Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like"Let's think step by step"to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the"Let's think step by step"prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot
Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Graham Neubig
- Caiming Xiong
- Yejin Choi
In Natural Language Processing
- Percy Liang
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Jonathan Berant
- Caiming Xiong
- Yejin Choi
- Hannaneh Hajishirzi
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Xinyun Chen
- Jie Huang
- Huan Sun
- Sean Welleck
- Kevin Chen-Chuan Chang
- Pan Lu
Citations in the corpus
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “Existing study [57] have divided such a technique into two categories: Few-ShotCoT [56] and Zero-Shot-CoT [20].”
- A Survey of Deep Learning for Mathematical Reasoning (2022, positive): “A Graph Neural Network (GNN) is employed for geometry problem parsing in Zhang et al. (2022), taking advantage of its success in spatial reasoning.”
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (2022, methodology): “DSP introduces a number of composable functions that bootstrap training examples (DEMONSTRATE), gather information from a knowledge corpus (SEARCH), and generate grounded outputs (PREDICT), using them to systematically unify techniques from the retrieval-augmented NLP and the in-context learning lit”
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, methodology): “We manually examined these failures and classified them into six categories as shown in Figure 1 and discussed next.”
- Better Zero-Shot Reasoning with Role-Play Prompting (2023, methodology): “Following previous work (Kojima et al. 2022; Zhang et al. 2022), we use greedy decoding across all the experiments by setting the temperature to 0, making the results deterministic.”
- Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, neutral): “We also visualize the distribution of improvements across tokens in the training set.”
- A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, background): “Subsequently, researchers began to improve the reasoning of large language models from multiple different research perspectives, such as automatically generating CoT prompts (zero-shot reasoner) [59], generating multiple associated simple sub-tasks to complete a complex task [60], [61], building mul”
- RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Since the introduction of GPT-3 [73], prompt learning has evolved from traditional discrete [69] and continuous prompt construction [59], [70] to large-scale model-oriented in-context learning [32], instruction-tuning [74], [75], [76], and chain-of-thought approaches [77], [78], [79].”
- A Survey on In-context Learning (2022, methodology): “…et al., 2022), APE (Zhou et al., 2022c), Self-Instruct (Wang et al., 2022b) (§5.2.2) CoT (Wang et al., 2022b), Complex CoT (Fu et al., 2022), AutoCoT (Zhang et al., 2022b), Self-Ask (Press et al., 2022), MoT(Li and Qiu, 2023b), SuperICL(Xu et al., 2023b) iCAP (Wang et al., 2022a),…”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “To alleviate the burden of manual efforts, [33] introduced Auto-CoT, an approach that automates the process of acquiring k samples through the application of clustering techniques on a provided dataset.”
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2023, background): “, 2022), Auto-CoT (Zhang et al., 2022), Active-Prompting (Diao et al.”
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “…instruction or prompting methods , such as Chain-of-Thought (CoT) [123], zero-shot-CoT [61], and ReAct [126], have recently been developed to enhance the performance of LLMs on reasoning, arithmetic and decision making tasks [134, 118, 52, 73, 31, 103, 43, 64, 132, 46, 133, 105, 128, 25, 81, 109].”
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023, methodology): “Following Auto-CoT (Zhang et al., 2022), we use the public GPT-3 (Brown et al.”
- A Survey of Large Language Models (2023, neutral): “In order to boost the performance, Auto-CoT further divides the questions in the training set into different clusters and then chooses the questions that are closest to the centroid of each cluster, which is supposed to well represent the questions in the training set.”