Contextualized Word Representations for Reading Comprehension

By Shimi Salant, Jonathan Berant. 2017

42 citations recorded by the source corpus.

Reading a document and extracting an answer to a question about its content has attracted substantial attention recently. While most work has focused on the interaction between the question and the document, in this work we evaluate the importance of context when the question and document are processed independently. We take a standard neural architecture for this task, and show that by providing rich contextualized word representations from a large pre-trained language model as well as allowing the model to choose between context-dependent and context-independent word representations, we can obtain dramatic improvements and reach performance comparable to state-of-the-art on the competitive SQuAD dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, neutral): “…hold for question answering in general, so recent models for SQuAD are not necessarily general question answering models [Yu et al., 2018a, Hu et al., 2018, Wang et al., 2017a, Liu et al., 2017b, Huang et al., 2017, Xiong et al., 2018, Liu et al., 2017a, Pan et al., 2017, Salant and Berant, 2017].”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “ree key components is becoming standard practice, and various state-of-the-art or competitive architectures (Liu et al.,2018;Clark and Gardner,2017;Wang et al.,2017;Seo et al.,2017; Pan et al. ,2017 ;Salant and Berant 2018 Xiong et al.,2018) on SQuAD can be viewed as similar to our implemented model”
The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “RaSoR + TR + LM (Single) (Salant and Berant, 2018) https://worksheets.”
Language Model Pre-training for Hierarchical Document Representations (2019, unknown): “…to use unlabeled text to pre-train representations of individual words, e.g. (Mikolov et al., 2013), or ﬂat (relatively short) sequences of words, where each word representation is contextualized with respect to the sequence (Peters et al., 2017; 2018; Salant & Berant, 2018; Radford et al., 2018).”
Simple and Effective Semi-Supervised Question Answering (2018, methodology): “Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model (Salant and Berant, 2017; Peters et al., 2018; McCann et al., 2017).”
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension (2018, background): “Even further performance improvement is gained by using contextualized word representations for query and context (Salant and Berant, 2017; Peters et al., 2018; Yu et al., 2018).”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “Most of the papers that approach to solve SQuAD problem have utilized RNN based models [3, 5, 7, 9, 13, 14, 16, 17, 20, 22–24, 26, 29–31, 34–39, 41, 43] and they follow a similar chain of processes.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “…been evaluated in a wide range of NLP tasks, including sentiment analysis, question answering and classiﬁcation, textual entailment, semantic role labeling, reading comprehension, named entity extraction, and coreference resolution (Peters et al., 2018; Salant & Berant, 2018; McCann et al., 2017).”
Alternating Language Modeling for Cross-Lingual Pre-Training (2020, background): “…et al. 2019), have achieved impressive results on various natural language processing tasks such as question-answering (Min, Seo, and Hajishirzi 2017; Yang et al. 2019a), machine reading comprehension (Salant and Berant 2018; Yu et al. 2018) and natural language inference (Tay, Luu, and Hui 2018).”
Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension (2018, background): “Salant and Berant (2018) proposed to minimize (but not prohibit) the inﬂuence of question when modeling the document.”