Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

By Akari Asai, Hannaneh Hajishirzi. 2020

126 citations recorded by the source corpus.

Many natural language questions require qualitative, quantitative or logical comparisons between two entities or events. This paper addresses the problem of improving the accuracy and consistency of responses to comparison questions by integrating logic rules and neural models. Our method leverages logical and linguistic knowledge to augment labeled training data and then uses a consistency-based regularizer to train the model. Improving the global consistency of predictions, our approach achieves large improvements over previous methods in a variety of question answering (QA) tasks, including multiple-choice qualitative reasoning, cause-effect reasoning, and extractive machine reading comprehension. In particular, our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets. We advance state of the art by around 5-8% on WIQA and QuaRel and reduce consistency violations by 58% on HotpotQA. We further demonstrate that our approach can learn effectively from limited data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Reasoning Like Program Executors (2022, methodology): “, 2021), generated by a model-based generator (Asai and Hajishirzi, 2020), or synthesized via human-designed templates (Geva et al.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “In some more constrained domains, such as reasoning about quantitative comparisons (more than, less than, etc.), even relatively small numbers of synthesized training examples have proven useful [23], [24].”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “ing comprehension tasks regardless of paraphrase (Asai and Hajishirzi, 2020); making consistent assignments in coreference resolution (Denis and Baldridge, 2009; Chang et al.”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “, 2020), and to improve consistency (Asai and Hajishirzi, 2020), and zero-shot accuracy (Zhao et al.”
On the Advance of Making Language Models Better Reasoners (2022, methodology): “For example, Deng et al. (2021) proposed to train the language model with crawled data from the internet; Asai and Hajishirzi (2020) proposed a logic-guided data augmentation method to pre-train the language model; Reasoning via Prompting Gigantic Language Models.”
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021, background): “, (Ribeiro et al., 2019; Minervini and Riedel, 2018; Li et al., 2019; Asai and Hajishirzi, 2020), they have not solved the problem.”
Substructure Substitution: Structured Data Augmentation for NLP (2021, background): “3496 Another approach to example generation is to generate new examples based on predefined templates (Kafle et al., 2017; Asai and Hajishirzi, 2020), where the templates are designed following heuristic, and usually task-specific, rules.”
Large Language Models are Better Reasoners with Self-Verification (2022, methodology): “To solve these reasoning tasks, researchers have utilized pre-trained language reasoning models (Asai and Hajishirzi, 2020; Deng et al., 2021) or fine-tuned general LLMs (Cobbe et al.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, unknown): “Sa m pl in g Rules [24], [26], [69], [75], [102], [115], [116], [117] [74], [118], [119], [120] [121]”
Enriching a Model's Notion of Belief using a Persistent Memory (2021, background): “Others have trained models with (small) sets of examples with known constraints between them, and included an additional loss term reflecting inconsistency among set members during training (Minervini and Riedel, 2018; Li et al., 2019; Asai and Hajishirzi, 2020).”
Table Fact Verification with Structure-Aware Transformer (2020, methodology): “Pre-trained models like BERT are good at semantic-level understanding, but not symbolic reasoning (Geva et al., 2020; Asai and Hajishirzi, 2020).”