Flexible Generation of Natural Language Deductions

By Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, Greg Durrett. 2021

38 citations recorded by the source corpus.

An interpretable system for open-domain reasoning needs to express its reasoning process in a transparent form. Natural language is an attractive representation for this purpose — it is both highly expressive and easy for humans to understand. However, manipulating natural language statements in logically consistent ways is hard: models must cope with variation in how meaning is expressed while remaining precise. In this paper, we describe ParaPattern, a method for building models to generate deductive inferences from diverse natural language inputs without direct human supervision. We train BART-based models (Lewis et al., 2020) to generate the result of applying a particular logical operation to one or more premise statements. Crucially, we develop a largely automated pipeline for constructing suitable training examples from Wikipedia. We evaluate our models using out-of-domain sentence compositions from the QASC (Khot et al., 2020) and EntailmentBank (Dalvi et al., 2021) datasets as well as targeted perturbation sets. Our results show that our models are substantially more accurate and flexible than baseline systems. ParaPattern achieves 85% validity on examples of the ‘substitution’ operation from EntailmentBank without the use of any in-domain training data, matching the performance of a model fine-tuned for EntailmentBank. The full source code for our method is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

NaturalProver: Grounded Mathematical Proof Generation with Language Models (2022, background): “Related areas in NLP. Systematic reasoning in natural language (outside of math) has been studied with synthetic proofs [36, 40], single-step deductions [4], or entailment trees [8], which differ from proving real-world mathematical theorems.”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, background): “Other works explore how to train a model that simulates the symbolic reasoning process, such as logical transformation (Bostrom et al., 2021) and consistent generation of beliefs (Kassner et al., 2021; Dalvi et al., 2022).”
Natural Language Deduction through Search over Statement Compositions (2022, methodology): “Repetition Past work on natural language deduction (Bostrom et al., 2021) has identified an important failure mode of deduction models.”
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022, background): “Conditioning the retrieval of a passage on previously retrieved passages has been explored in the context of multi-hop QA (Zhao et al., 2021; Xiong et al., 2021), and multi-hop explanations (Valentino et al., 2021; Cartuyvels et al., 2020).”