Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

By Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi. 2022

237 citations recorded by the source corpus.

Pre-trained language models (LMs) struggle with consistent reasoning; recently, prompting LMs to generate explanations that self-guide the inference has emerged as a promising direction to amend this. However, these approaches are fundamentally bounded by the correctness of explanations, which themselves are often noisy and inconsistent. In this work, we develop Maieutic Prompting, which aims to infer a correct answer to a question even from the unreliable generations of LM. Maieutic Prompting induces a tree of explanations abductively (e.g. X is true, because ...) and recursively, then frames the inference as a satisfiability problem over these explanations and their logical relations. We test Maieutic Prompting for true/false QA on three challenging benchmarks that require complex commonsense reasoning. Maieutic Prompting achieves up to 20% better accuracy than state-of-the-art prompting methods, and as a fully unsupervised approach, performs competitively with supervised models. We also show that Maieutic Prompting improves robustness in inference while providing interpretable rationales.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Large Language Models Can Self-Improve (2022, neutral): “For example, Ye and Durrett (2022) calibrate model predictions based on the reliability of the explanations, Jung et al. (2022) show that inducing a tree of explanations and inferring the satisfiability of each explanation can further help judge the correctness of explanations.”
Rationale-Augmented Ensembles in Language Models (2022, methodology): “…pretrained large language models has been achieved by expanding prompt exemplars with rationales, delivering successes in a variety of natural language reasoning tasks (Wei et al., 2022; Chowdhery et al., 2022; Lampinen et al., 2022; Creswell et al., 2022; Kojima et al., 2022; Jung et al., 2022).”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, background): “…generating sub-questions and answering them sequentially in least-to-most complexity order (Zhou et al., 2022), integration with symbolic modules (Jung et al., 2022), selection inference (Creswell et al., 2022), coupling in-context learning with sparse fine-tuning (Zelikman et al., 2022), and…”
Tree of Thoughts: Deliberate Problem Solving with Large Language Models (2023, background): “This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM’s own thoughts, and there is no reﬂection or voting steps.”
Reasoning with Language Model Prompting: A Survey (2022, background): “, 2022), commonsense (Jung et al., 2022; Liu et al., 2022b), symbolic (Zhou et al.”
Teaching Algorithmic Reasoning via In-context Learning (2022, background): “Moreover, various prompting strategies have shown signiﬁcant potential in solving certain types of reasoning problems (Jung et al., 2022; Zhou et al., 2022; Wei et al., 2022b; Kojima et al., 2022).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “One direction is to explore new inference-time (Jung et al., 2022; Mitchell et al., 2022) or fine-tuning (Anil et al.”
Discovering Latent Knowledge in Language Models Without Supervision (2022, background): “Particularly relevant are methods that also leverage unsupervised consistency properties, such as Jung et al. (2022); Zhou et al. (2022).”
Large Language Models as Analogical Reasoners (2023, background): “…with more structured algorithms and search methods (Khot et al., 2022; Drozdov et al., 2022; Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as longer-horizon action and planning (Yao et al., 2022; Hao et al., 2023; Park et al., 2023).”
Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “Some formats are designed-for or evaluated-on a narrow task type, model type, or model size [Wei et al., 2022a, Jung et al., 2022].”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “Evaluating Plausibility of CoT Explanations Existing evaluations of CoT explanations ﬁnd serious ﬂaws, including contradictions and mathematical errors (Uesato et al., 2022; Jung et al., 2022; Ye and Durrett, 2022; Golovneva et al., 2023).”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “Aside from methods that make reasoning explicit in a linear chain manner, another set of work produce recursive structures of reasoning, through either backward chaining (Dalvi et al., 2022; Jung et al., 2022) or forward chaining (Bostrom et al., 2022).”
Reasoning with Language Model is Planning with World Model (2023, methodology): “In the domains of mathematical reasoning, such as GSM8K (Cobbe et al., 2021) and logical inference exemplified by PrOntoQA (Saparov and He, 2022), RAP also consistently improves over strong baselines, including CoT, least-to-most prompting, and their self-consistency variants.”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, background): “While large-scale pre-training allows models to acquire sufficient knowledge to solve complex tasks (Bommarito et al., 2023; Brahman et al., 2023; Jung et al., 2022), recent works suggest that their full capability is elicited from aligning the model knowledge with additional fine-tuning – e.g.…”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “Due to the convincing nature of this generated text, however, users may not perform appropriate due diligence to determine its accuracy.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, unknown): “Recent efforts have been made to investigate non-linear reasoning structures such as trees(Jung et al., 2022; Zhu et al., 2023; Xu, 2023; Xie et al., 2023; Yao et al., 2023; Hao et al., 2023).”