Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango

By Aman Madaan, Amir Yazdanbakhsh. 2022

153 citations recorded by the source corpus.

The past decade has witnessed dramatic gains in natural language processing and an unprecedented scaling of large language models. These developments have been accelerated by the advent of few-shot techniques such as chain of thought (CoT) prompting. Specifically, CoT pushes the performance of large language models in a few-shot setup by augmenting the prompts with intermediate steps. Despite impressive results across various tasks, the reasons behind their success have not been explored. This work uses counterfactual prompting to develop a deeper understanding of CoT-based few-shot prompting mechanisms in large language models. We first systematically identify and define the key components of a prompt: symbols, patterns, and text. Then, we devise and conduct an exhaustive set of experiments across four different tasks, by querying the model with counterfactual prompts where only one of these components is altered. Our experiments across three models (PaLM, GPT-3, and CODEX) reveal several surprising findings and brings into question the conventional wisdom around few-shot prompting. First, the presence of factual patterns in a prompt is practically immaterial to the success of CoT. Second, our results conclude that the primary role of intermediate steps may not be to facilitate learning how to solve a task. The intermediate steps are rather a beacon for the model to realize what symbols to replicate in the output to form a factual answer. Further, text imbues patterns with commonsense knowledge and meaning. Our empirical and qualitative analysis reveals that a symbiotic relationship between text and patterns explains the success of few-shot prompting: text helps extract commonsense from the question to help patterns, and patterns enforce task understanding and direct text generation.

Subfields: Artificial Intelligence, Information Systems, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Prompting & Reasoning

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “Madaan and Yazdanbakhsh (2022) show that, while simply us-ing a random rationale in the demonstrations (e.g., pairing with a rationale from a different example) signiﬁcantly degrades the performance, other types of counterfactual rationales (e.g., wrong equations) do not degrade the performance as…”
PAL: Program-aided Language Models (2022, methodology): “Nevertheless, all previous approaches suffer from inaccuracy in arithmetic calculation and incorrect reasoning (Lewkowycz et al., 2022; Hendrycks et al., 2021; Madaan & Yazdanbakhsh, 2022).”
A Survey of Large Language Models (2023, unknown): “, the rest of tokens that are not symbols or patterns) [259].”
Measuring Faithfulness in Chain-of-Thought Reasoning (2023, background): “Madaan & Yazdanbakhsh (2022) investigate CoT via counterfactual prompting and ﬁnd that some aspects of the prompt are less important than others for the ﬁnal answer reached.”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “Additionally, they exhibit weaknesses in numerical reasoning, an essential skill in numerous real-life applications [14, 36, 41, 28, 51, 13].”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “Effects of Perturbations on CoT A line of recent work (Ye et al., 2022; Madaan and Yazdanbakhsh, 2022; Wang et al., 2023) investigates perturbing CoT demonstrations in a few-shot prompt, e.”
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (2024, methodology): “…of generating thought steps extends to zero-shot CoT prompting (Kojima et al., 2022), where the LLMs are only simply instructed with the prompt “let’s think step by step”, and to even using incorrect reasoning steps in the few-shot examples (Wang et al., 2022a; Madaan & Yazdanbakhsh, 2022).”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “Madaan and Yazdanbakhsh (2022) investigates the importance of different components of the demonstrated CoT rationales by changing them to be counterfactual.”
Explainability for Large Language Models: A Survey (2023, methodology): “Hernandez et al. [2022] find that the performance of an 800M parameter model can degrade to that of a 400M parameter model by only repeating 10% of the training data.”
Think before you speak: Training Language Models With Pause Tokens (2023, background): “Other empirical works (Turpin et al., 2023; Wang et al., 2023a; Madaan & Yazdanbakhsh, 2022) have shown that the generated intermediate reasoning steps can be unfaithful, not representing the model’s true reasoning process.”
Training Large Language Models to Reason in a Continuous Latent Space (2024, unknown): “While previous work has attempted to fix these problems by prompting LLMs to generate succinct reasoning chains (Madaan and Yazdanbakhsh, 2022), or performing additional reasoning before generating some critical tokens (Zelikman et al., 2024), these solutions remain constrained within the language…”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “…can provide human-readable explanations of how problems are solved (Joshi et al., 2023; Lanham et al., 2023), but most frequently it is invoked to improve an LLM’s ability to answer complex questions via intermediate computation (Madaan & Yazdanbakhsh, 2022; Wang et al., 2023a; Dziri et al., 2023).”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “In particular, Madaan & Yazdanbakhsh (2022) conclude that the correctness of numbers and equations in chain-of-thought prompts does not play a key role in model performance, but using wrong entities and removing either equations or text explanation in the reasoning steps drastically hamper the…”
Larger language models do in-context learning differently (2023, background): “Along the same lines, Madaan & Yazdanbakhsh (2022) and Wang et al. (2022) show that for chain-of-thought prompting (Wei et al., 2022c), logically-incorrect prompts do not hurt performance on multi-step reasoning tasks.”
Large Language Models as Optimizers (2023, background): “LLMs are shown to be sensitive to the prompt format (Zhao et al., 2021; Lu et al., 2021; Wei et al., 2023; Madaan & Yazdanbakhsh, 2022); in particular, semantically similar prompts may have drastically different performance (Kojima et al., 2022; Zhou et al., 2022b; Zhang et al., 2023), and the…”
Language Models can Solve Computer Tasks (2023, background): “This discovery has led to the emergence of a new line of research where reasoning capabilities are used to address tasks beyond reasoning [61, 22], or enhance reasoning proficiency [26, 28, 54, 29, 62, 40, 58, 51, 12].”