Improving Machine Reading Comprehension with General Reading Strategies

By Kai Sun, Dian Yu, Dong Yu, Claire Cardie. 2018

118 citations recorded by the source corpus.

Reading strategies have been shown to improve comprehension levels, especially for readers lacking adequate prior knowledge. Just as the process of knowledge accumulation is time-consuming for human readers, it is resource-demanding to impart rich general domain knowledge into a deep language model via pre-training. Inspired by reading strategies identified in cognitive science, and given limited computational resources - just a pre-trained model and a fixed number of training instances - we propose three general strategies aimed to improve non-extractive machine reading comprehension (MRC): (i) BACK AND FORTH READING that considers both the original and reverse order of an input sequence, (ii) HIGHLIGHTING, which adds a trainable embedding to the text embedding of tokens that are relevant to the question and candidate answers, and (iii) SELF-ASSESSMENT that generates practice questions and candidate answers directly from the text in an unsupervised manner. By fine-tuning a pre-trained language model (Radford et al., 2018) with our proposed strategies on the largest general domain multiple-choice MRC dataset RACE, we obtain a 5.8% absolute increase in accuracy over the previous best result achieved by the same pre-trained model fine-tuned on RACE without the use of strategies. We further fine-tune the resulting model on a target MRC task, leading to an absolute improvement of 6.2% in average accuracy over previous state-of-the-art approaches on six representative non-extractive MRC datasets from different domains (i.e., ARC, OpenBookQA, MCTest, SemEval-2018 Task 11, ROCStories, and MultiRC). These results demonstrate the effectiveness of our proposed strategies and the versatility and general applicability of our fine-tuned models that incorporate these strategies. Core code is available at https://github.com/nlpdata/strategy/.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Option Comparison Network for Multiple-choice Reading Comprehension (2019, background): “…models have been proposed for MCRC and achieve promising re-sults in recent years (Chen et al., 2016; Yin et al., 2016; Trischler et al., 2016; Dhingra et al., 2017; Tay et al., 2018; Parikh et al., 2018; Zhu et al., 2018; Wang et al., 2018; Xu et al., 2017; Sun et al., 2018; Zhang et al., 2019).”
Improving Question Answering with External Knowledge (2019, methodology): “…dataset side, our work primarily focuses on multiple-choice examination datasets designed by educational experts (Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019) since questions from these datasets are generally clean, error-free, and challenging (Sun et al., 2019).”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “…is essential and indeed extremely helpful (Devlin et al. 2019; Yang et al. 2019) but at the same time it is time-consuming and resource-demanding to impart massive amounts of general knowledge from external corpora into a deep language model via pre-training (Sun et al. 2019; Zhang et al. 2019b).”
Transformers as Soft Reasoners over Language (2020, neutral): “, 2018] and improve question-answering [Sun et al., 2018].”
QASC: A Dataset for Question Answering via Sentence Composition (2019, methodology): “We primarily use BERT-models ﬁne-tuned on other QA datasets and with retrieved sentences as context, similar to prior state-of-the-art models on MCQ datasets (Sun et al. 2018; Pan et al. 2019).”
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (2019, methodology): “CWQ (Talmor and Berant, 2018c): Crowd-sourcing workers were shown compositional formal queries against Freebase and were asked to re-phrase them in natural language.”
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions (2019, methodology): “BERT-PFT (no knowledge): BERT ﬁrst ﬁne-tuned (“pre-ﬁne-tuned”) on the RACE dataset (Lai et al., 2017; Sun et al., 2019), and then ﬁne-tuned on Q UA RT Z (questions only, no K , both train and test).”
FreeLB: Enhanced Adversarial Training for Natural Language Understanding (2019, methodology): “Following Sun et al. (2018), we ﬁrst ﬁnetune the RoBERTa model on the RACE dataset (Lai et al., 2017).”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, methodology): “Curriculum Fine-Tuning Following earlier work on multi-step fine-tuning (Sun et al., 2019), we first finetune on the large (87866 qs) RACE training set (Lai et al.”
Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (2018, methodology): “A similar outcome has been reported in Jansen et al. (2017); Khashabi et al. (2017) where essential term extraction methods have shown around 2%-4% gain compared with TF-IDF models and struggle to obtain further improvement on SQA tasks.”
Careful Selection of Knowledge to Solve Open Book Question Answering (2019, background): “, 2018) which perform QA using fine tuned language model and the works of (Sun et al., 2018; Zhang et al., 2018) which performs QA using external knowledge.”
Explainable Natural Language Reasoning via Conceptual Unification (2020, methodology): “The SWCU model is still outperformed by reading strategies that adopt pre-training on exter-nal question answering datasets (Sun et al., 2019), which, however, do not produce explanations for their predicted answers.”
“You Are Grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020, methodology): “Conversely, Winogrande models are mostly insulated from this effect.”
FreeLB: Enhanced Adversarial Training for Language Understanding (2019, methodology): “Following Sun et al. (2018), we ﬁrst ﬁnetune the RoBERTa model on the RACE dataset (Lai et al., 2017).”
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering (2020, unknown): “We stick to our focus of encoding structured KG and therefore do not compare our models with those (Pan et al., 2019; Zhang et al., 2018; Sun et al., 2019; Banerjee et al., 2019) augmented by other text-form external knowledge ( e.g. , Wikipedia).”
Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian (2019, methodology): “…extractive, e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g., multiple-choice questions (Richardson et al., 2013; Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019a).”
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension (2019, methodology): “First, we follow previous work (Sun et al., 2019b) for English question answering tasks.”