Improving Question Answering with External Knowledge

By Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, Dong Yu. 2019

68 citations recorded by the source corpus.

We focus on multiple-choice question answering (QA) tasks in subject areas such as science, where we require both broad background knowledge and the facts from the given subject-area reference corpus. In this work, we explore simple yet effective methods for exploiting two sources of external knowledge for subject-area QA. The first enriches the original subject-area reference corpus with relevant text snippets extracted from an open-domain resource (i.e., Wikipedia) that cover potentially ambiguous concepts in the question and answer options. As in other QA research, the second method simply increases the amount of training data by appending additional in-domain subject-area instances. Experiments on three challenging multiple-choice science QA tasks (i.e., ARC-Easy, ARC-Challenge, and OpenBookQA) demonstrate the effectiveness of our methods: in comparison to the previous state-of-the-art, we obtain absolute gains in accuracy of up to 8.1%, 13.0%, and 12.8%, respectively. While we observe consistent gains when we introduce knowledge from Wikipedia, we find that employing additional QA training instances is not uniformly helpful: performance degrades when the added instances exhibit a higher level of difficulty than the original training data. As one of the first studies on exploiting unstructured external knowledge for subject-area QA, we hope our methods, observations, and discussion of the exposed limitations may shed light on further developments in the area.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Memory-assisted prompt editing to improve GPT-3 after deployment (2022, methodology): “Extensive prior work has used retrievals from a text corpus to aid QA, e.g., Pan et al. (2019); Guu et al. (2020), or retrievals of prior QA pairs for nearest-neighbor QA (Khan-delwal et al., 2020).”
QASC: A Dataset for Question Answering via Sentence Composition (2019, methodology): “We primarily use BERT-models ﬁne-tuned on other QA datasets and with retrieved sentences as context, similar to prior state-of-the-art models on MCQ datasets (Sun et al. 2018; Pan et al. 2019).”
Towards Debiasing Sentence Representations (2020, background): “More recently, sentence representations such as ELMo [20], BERT [8], and GPT [21] have become the preferred choice for encoding text [30, 17, 31, 23, 26].”
Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian (2019, methodology): “, 2019b), linked entities (Pan et al., 2018), concatenation and reformulation of passages and questions (Simov et al.”
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (2019, background): “Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they ﬁrst retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, background): “Existing works (Pan et al., 2019; Ye et al., 2019; Petroni et al., 2019; Bosselut et al., 2019) study pretrained LMs’ potential as latent knowledge bases.”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, background): “Note that some of the works discussed here transfer knowledge from external datasets into the QA task they address (Chung et al., 2017; Sun et al., 2018; Pan et al., 2019; Min et al., 2017; Qiu et al., 2018; Chen et al., 2017).”
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering (2020, background): “Recent success of pre-trained LMs motivates many (Pan et al., 2019; Ye et al., 2019; Zhang et al., 2018; Li et al., 2019; Banerjee et al., 2019) to probe LMs’ potential as latent knowledge bases.”
Segatron: Segment-Aware Transformer for Language Modeling and Understanding (2020, unknown): “Results of BERT-large are from Pan et al. (2019).”