Dissecting Contextual Word Embeddings: Architecture and Representation

By Matthew E. Peters, Mark Neumann, Luke S. Zettlemoyer, Wen-tau Yih. 2018

492 citations recorded by the source corpus.

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks. Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers. Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “ v2.0 Test F1 to 83.1 (5.1 point absolute improvement). 1 Introduction Language model pre-training has been shown to be effective for improving many natural language processing tasks (Dai and Le,2015;Peters et al., 2018a;Radford et al.,2018;Howard and Ruder, 2018). These include sentence-level tasks”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “This stands in contrast to the approach originally used for ELMo (Peters et al., 2018b) and for ear-”
Transfer Learning in Natural Language Processing (2019, background): “…like auto-encoding/skip-thoughts models (Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) and the current trend of language model-based approaches (Dai and Le, 2015; Peters et al., 2018a; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “Howard and Ruder (2018), Peters et al. (2018a), Radford et al. (2018), and Devlin et al. (2019) establish that encoders pretrained on variants of the language modeling task can be reused to yield strong performance on downstream NLP tasks.”
Language Models as Knowledge Bases? (2019, methodology): “The first version of GPT (Radford et al., 2018) has been trained on the Book Corpus (Zhu et al., 2015) containing 7000 books.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “Large pre-trained language models (LM) have revolutionized the field of natural language processing in the last few years (Dai and Le, 2015; Peters et al., 2018a; Yang et al., 2019; Radford et al., 2019; Devlin et al., 2019).”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “Studies have shown that in a deep LM, lower layers tend to focus more on local syntax, while the upper layers usually induce more semantic content [35].”
BERT Rediscovers the Classical NLP Pipeline (2019, methodology): “Pre-trained sentence encoders such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have rapidly improved the state of the art on many NLP tasks, and seem poised to displace both static word embeddings (Mikolov et al., 2013) and discrete pipelines (Manning et al., 2014) as the basis…”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “Recent work has explored many token-level properties of these representations, such as their ability to capture part-of-speech tags (Blevins et al., 2018; Belinkov et al., 2017b; Shi et al., 2016), morphology (Belinkov et al., 2017a;b), or word-sense disambiguation (Peters et al., 2018a).”
Designing and Interpreting Probes with Control Tasks (2019, methodology): “…obtained high accuracy on tasks requiring part-of-speech and morphological information (Belinkov et al., 2017), syntactic and semantic information (Peters et al., 2018b; Tenney et al., 2019), among other properties (Conneau et al., 2018), providing evidence that deep representations trained on…”
A Structural Probe for Finding Syntax in Word Representations (2019, methodology): “Our work extends the literature on linguistic probes, found at least in (Peters et al., 2018b; Belinkov et al., 2017; Blevins et al., 2018; Hupkes et al., 2018).”
How Multilingual is Multilingual BERT? (2019, background): “However, cross-script transfer is less accurate for other pairs, such as English and Japanese, indicating that M-B ERT ’s multilingual representation is not able to generalize equally well in all cases.”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “In contemporaneous work, Tenney et al. (2019) evaluate CoVe (McCann et al., 2017), ELMo (Peters et al., 2018a), the OpenAI Transformer (Rad-ford et al., 2018), and BERT (Devlin et al., 2018) on a variety of sub-sentence linguistic analysis tasks.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Prior work in NLP also suggests that deeper layers in text-based Transformers attend to more complex properties [82] and encode higher-level representations [34, 54, 56, 74].”
What Does BERT Learn about the Structure of Language? (2019, methodology): “Following Peters et al. (2018), for a token sequence s i , . . . , s j , we compute the span representation s ( s i ,s j ) ,l at layer l by concatenating the ﬁrst ( h s i ,l ) and last hidden vector ( h s j ,l ), along with their element-wise product and difference.”
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, methodology): “Similarly, recent works—including our own—have proposed meth-ods that produce different embeddings for the same word depending on its contextual usage (Peters et al., 2018a; Akbik et al., 2018; Devlin et al., 2018).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “…neural representations of words and sentences from large generic corpora has already shown great success in transferring syntactic and, to some extent, semantic information to downstream tasks in natural language understanding [ Peters et al. , 2018b; Goldberg, 2019; Tenney et al. , 2019 ] .”