Learned in Translation: Contextualized Word Vectors

By Bryan McCann, James Bradbury, Caiming Xiong, Richard Socher. 2017

941 citations recorded by the source corpus.

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “…of unsupervised pre-training is that there is a nearly unlimited amount of data available, there has also been work showing effective transfer from supervised tasks with large datasets, such as natural language inference (Conneau et al., 2017) and machine translation (McCann et al., 2017).”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “Pre-training on other tasks such as machine translation has also been studied (McCann et al., 2017).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “In addition to biLM-based representations, McCann et al. (2017) learned contextualized vectors with a neural machine translation system (CoVe).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “, 2018), labeled data (Conneau and Kiela, 2018; McCann et al., 2017), and combinations of these (Collobert et al.”
Deep Contextualized Word Representations (2018, methodology): “For tasks where direct comparisons are possible, ELMo outperforms CoVe (McCann et al., 2017), which computes contextualized representations using a neural machine translation encoder.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “On IMDb, we reduce the error dramatically by 43.9% and 22% with regard to CoVe and the state-of-the-art respectively.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “Pretraining methods have been designed with different training objectives, including language modeling (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018), machine translation (McCann et al., 2017), and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019).”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “Unsupervised representation learning has been highly successful in the domain of natural language processing [7, 22, 27, 28, 10].”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Although early work wrestled with the complexity of natural language when using topic model and n-gram representations, improvements in deep contextual representation learning suggest we now have the tools to effectively leverage this abundant source of supervision (McCann et al., 2017).”
Experience Grounds Language (2020, background): “learned from text capture a great deal of information about meaning in and out of context (Collobert and Weston, 2008; Turian et al., 2010; Mikolov et al., 2013; McCann et al., 2017).”
GLTR: Statistical Detection and Visualization of Generated Text (2019, background): “The success of pretrained language models for natural language understanding (McCann et al., 2017; Devlin et al., 2018; Peters et al., 2018) has led to a race to train unprecedentedly large language models (Radford et al.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al., 2015; Logeswaran & Lee, 2018; Le & Mikolov, 2014), as well as context sensitive features from the NMT model (McCann et al., 2017) and ELMo (Peters et al., 2018).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Much work prior to GLUE has demonstrated that training neural models with large amounts of available supervision can produce representations that effectively transfer to a broad range of NLP tasks (Collobert and Weston, 2008; Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Conneau and Kiela”
Massively Multilingual Neural Machine Translation (2019, background): “duce the performance degradation when increasing the number of languages, or using such models for multilingual transfer learning (McCann et al., 2017; Eriguchi et al., 2018; Artetxe and Schwenk, 2018).”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “This stands in contrast to the approach used for methods like ELMo (Peters et al. ) and CoVe (McCann et al., 2017) and for earlier sentence-to-vector methods like (Subramanian et al., 2018), in which a sentence encoder component is pretrained and then attached to a target task model as an ﬁxed input”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Intermediate representations from supervised machine translation models improve performance on question answering, sentiment analysis, and natural language inference (McCann et al., 2017).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “They are also used as a means of learning word vectors (Mikolov et al., 2013), document vectors (Kiros et al., 2015), or contextualized word vectors (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018) for transfer learning.”