Deep Contextualized Word Representations
By Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton C. T. Lee, Luke S. Zettlemoyer. North American Chapter of the Association for Computational Linguistics, 2018
12,212 citations recorded by the source corpus.
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/1802.05365
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Quoc V. Le
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
- Jianfeng Gao
- Percy Liang
- Noah A. Smith
- Caiming Xiong
In Representation & Self-Supervised Learning
- Sebastian Ruder
- Yoshua Bengio
- Ivan Vulic
- James R. Glass
- Graham Neubig
- Jianfeng Gao
- Quoc V. Le
- Yonatan Belinkov
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “BERTLARGE outperforms the authors’ baseline ESIM+ELMo system by +27.1%.”
- Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “The ELMo accounts for about 5% of the performance degradation, which clearly shows the effectiveness of language model.”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “More recent work trains a Bi-LSTM sentence encoder to do language modeling and then incorporates its context-sensitive representations into supervised models (Dai and Le, 2015; Peters et al., 2018).”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “With the 2-layer LSTM ELMo representations, it is state-of-the-art for SNLI (Peters et al., 2018).”
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “…300d GloVe vectors from Common Crawl (Pennington et al., 2014), 300d Numberbatch vectors retrofitted using ConceptNet relations (Speer et al., 2017), and 1024d ELMo contextual representations that show improvement on a variety of NLP tasks, including standard NLI (Peters et al., 2018).”
- Neural Network Acceptability Judgments (2018, methodology): “It also uses contexualized word embeddings inspired by ELMo (Peters et al., 2018).”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “We confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, but still leave room for improvement.”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Specifically, Peters et al. (2018) require engineered custom architectures, while we show state-of-the-art performance with the same basic architecture across a range of tasks.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Early results on transfer learning for NLP leveraged recurrent neural networks [Peters et al., 2018; Howard and Ruder, 2018], but it has recently become more common to use models based on the “Transformer” architecture [Vaswani et al., 2017].”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “Self-training methods such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2019), XLM (Lample and Conneau, 2019), and XLNet (Yang et al., 2019) have brought significant performance gains, but it can be challenging to determine which aspects of the methods contribute th”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “Unsupervised representation learning has been highly successful in the domain of natural language processing [7, 22, 27, 28, 10].”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “Clark and Gardner (2018) proposed to use a global normalization over all the span corresponding to the answer, which was later applied to BERT based models (Wang et al., 2019).”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “, 2013a;b), sentences (Kiros et al., 2015; Peters et al., 2018), and paragraphs (Radford et al.”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “The NLP landscape has recently been revolutionized by language models (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020, inter alia ).”
- How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “This network is then fine-tuned (performs transfer learning) on the benchmark’s supervised tasks, each of which include a much smaller number of training examples than the pretraining corpus (Howard and Ruder, 2018; Peters et al., 2018).”
- Experience Grounds Language (2020, methodology): “…energy was directed at finding structure (e.g. syntax trees), this has been largely over-shadowed by the runaway success of unstructured, fuzzy representations—from dense word vectors (Mikolov et al., 2013) to contextualized pretrained representations (Peters et al., 2018b; Devlin et al., 2019b).”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “The popularity of this form of “transfer learning” is attributable to its empirical success on many NLP tasks (Peters et al., 2018; Devlin et al., 2018; Yang et al., 2019; Lan et al., 2019; Raffel et al., 2019).”