Complexity-Based Prompting for Multi-Step Reasoning

By Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, Tushar Khot. 2022

629 citations recorded by the source corpus.

We study the task of prompting large-scale language models to perform multi-step reasoning. Existing work shows that when prompted with a chain of thoughts (CoT), sequences of short sentences describing intermediate reasoning steps towards a final answer, large language models can generate new reasoning chains and predict answers for new inputs. A central question is which reasoning examples make the most effective prompts. In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning. We show that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, achieve substantially better performance on multi-step reasoning tasks over strong baselines. We further extend our complexity-based criteria from prompting (selecting inputs) to decoding (selecting outputs), where we sample multiple reasoning chains from the model, then choose the majority of generated answers from complex reasoning chains (over simple chains). When used to prompt GPT-3 and Codex, our approach substantially improves multi-step reasoning accuracy and achieves new state-of-the-art (SOTA) performance on three math benchmarks (GSM8K, MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared with existing example selection schemes like manual tuning or retrieval-based selection, selection based on reasoning complexity is intuitive, easy to implement, and annotation-efficient. Further results demonstrate the robustness of performance gains from complex prompts under format perturbation and distribution shift.

Subfields: Artificial Intelligence, Information Systems, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Prompting & Reasoning

Citations in the corpus

Multimodal Chain-of-Thought Reasoning in Language Models (2023, positive): “However, existing studies related to CoT reasoning are largely isolated in the language modality (Wang et al., 2022b; Zhou et al., 2022; Lu et al., 2022b; Fu et al., 2022), with little consideration of multimodal scenarios.”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, background): “Most of them attempted to make CoT comprehensive and fine-grained to present the step-by-step reasoning solutions (Nye et al., 2021; Fu et al., 2023; Zhou et al., 2023b; Khot et al., 2023; Imani et al., 2023; Miao et al., 2023).”
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, methodology): “An important line of work in this area is Chain-of-Thought (CoT) prompting which elicits the reasoning ability of models by prompting them to imitate the step-by-step reasoning process of humans (Wei et al., 2022b; Kojima et al., 2022; Fu et al., 2022; Zhou et al., 2022a).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “For a detailed explanation of the training pipeline, refer to Appendix A.6 also achieves remarkable performance on many reasoning benchmarks, which generates several possible answers from the model and selects the correct one based on majority vote (Fu et al., 2022).”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, neutral): “There is evidence that using additional calls to LLMs can help (Du et al., 2023; Yao et al., 2023; Besta et al., 2023; Chen et al., 2024), but these methods use significantly increased computation, and careful benchmarking sometimes reveals that naive techniques are as good as iterative ones…”
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2023, methodology): “Chen et al., 2023; Huang et al., 2022; Yao et al., 2022).”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, methodology): “The accuracy of Complexity-based CoT [20] for answering the rephrased question by GPT-3.”
Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “, 2023), and complex reasoning (Wei et al., 2022; Fu et al., 2022) in a zeroshot/few-shot setting, it is expected that LLMs can further empower VLPMs by powerful language abilities.”
A Survey on In-context Learning (2022, background): “…Instruction Induction (Honovich et al., 2022), APE (Zhou et al., 2022c), Self-Instruct (Wang et al., 2022b) (§5.2.2) CoT (Wang et al., 2022b), Complex CoT (Fu et al., 2022), AutoCoT (Zhang et al., 2022b), Self-Ask (Press et al., 2022), MoT(Li and Qiu, 2023b), SuperICL(Xu et al., 2023b) iCAP (Wang…”
A Survey of Large Language Models (2023, positive): “of LLMs [250], which can result in higher accuracy in generating correct answers.”
Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions (2023, methodology): “Despite the recent advancements in LLM research, such as self-consistency and group voting techniques [6, 20], as well as incorporating external expert models and APIs to enhance LLMs [9, 16], it has not been reported that LLMs like GPT-4 can benefit from additional opinions in the same way humans d”
Improving large language models for clinical named entity recognition via prompt engineering (2024, positive): “39 Second, our few-shot learning approaches were relatively simple, and we plan to investigate other approaches such as the chain-of-thoughts method, 40–42 hoping to yield better results.”
Towards Reasoning in Large Language Models: A Survey (2022, methodology): “Chain of thought elicits “reasoning” of LLMs.”
Large Language Models are Better Reasoners with Self-Verification (2022, background): “This approach has been shown the advance performances on several challenging NLP tasks, even when using only a few or no training samples (Madaan et al., 2022; Saparov and He, 2022; Fu et al., 2022).”