Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

By Sandeep Subramanian, Adam Trischler, Yoshua Bengio, C. Pal. 2018

339 citations recorded by the source corpus.

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “GenSen (Subramanian et al., 2018): a sequence-to-sequence model trained on a va-”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model [Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017].”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “…contrast to the approach used for methods like ELMo (Peters et al. ) and CoVe (McCann et al., 2017) and for earlier sentence-to-vector methods like (Subramanian et al., 2018), in which a sentence encoder component is pretrained and then attached to a target task model as an ﬁxed input layer that…”
Transfer Learning in Natural Language Processing (2019, background): “Last but not least, we will highlight the use of multi-task learning for pretraining (Subramanian et al., 2018; Cer et al., 2018; Devlin et al., 2018).”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “Sentence embedding methods Such methods learn sentence representations via different pre-training objectives such as previous/next sentence prediction (Kiros et al., 2015; Logeswaran and Lee, 2018), NLI (Conneau et al., 2017), or a combination of objectives (Subramanian et al., 2018).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “Luong et al. (2016) show promising results combining translation and parsing; Subramanian et al. (2018) beneﬁt from multitask learning in sentence-to-vector encoding; and Bingel and Søgaard (2017) and Changpinyo et al. (2018) offer studies of when multitask learning is helpful for lower-level NLP…”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “While these fixed-size sentence embedding methods have been outperformed by their supervised counterparts (Conneau et al., 2017; Subramanian et al., 2018), some recent developments have shown that pretrained language models can also transfer very well, either when the hidden states of the model are…”
Knowledge Enhanced Contextual Word Representations (2019, methodology): “Then, we extracted the gloss for each synset and used an off-the-shelf state-of-the-art sentence embedding method (Subramanian et al., 2018) to produce 2048-dimensional vectors.”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “…compare our sentence embedding method to previous approaches including BERT (Mean) (Devlin et al., 2018), InferSent (Conneau et al., 2017), GenSen (Subramanian et al., 2018), USE (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), uSIF (Ethayarajh, 2018a), Charagram (Wieting and…”
On Measuring Social Biases in Sentence Encoders (2019, methodology): “GenSen: A 2048-dimensional BiLSTM jointly trained on MultiNLI, SNLI, next sentence prediction, translation, and constituency parsing, concatenated to a similar BiLSTM trained without parsing; denoted “+STN +Fr +De +NLI +L +STP +Par” in Subramanian et al. (2018).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “…ranging from machine translation (Sutskever et al., 2014; Cho et al., 2014) to entailment detection (Williams et al., 2018), spurring the quest for “universal embeddings” trained once and used in a variety of applications (e.g., Kiros et al., 2015; Con-neau et al., 2017; Subramanian et al., 2018).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, positive): “This was recently extended to multitask learning, combining different training objectives like that of skip-thought, NLI and machine translation (Cer et al., 2018; Subramanian et al., 2018).”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, methodology): “To facilitate the learning process, we also adopted the attention mechanism that enables the Decoder to condition every decoding step on the last hidden state of the Encoder [28], in other words, the Decoder can refer to h T when generating every symbol y t of the output sequence y .”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “To adapt ExStream and SLDA for VQA, we use a variant of the linear VQA model in [34], which concatenates ResNet-101 image features to question features extracted from a universal sentence encoder [73] and then trains a linear classifier.”
Assessing Composition in Sentence Vector Representations (2018, background): “As part of this effort, we will expand to more comprehensive testing of a diverse range of sentence embedding systems (Bowman et al., 2016; Subramanian et al., 2018).”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “…that can be used without finetuning for downstream tasks (Le & Mikolov, 2014; Kiros et al., 2015; Wieting et al., 2016; Hill et al., 2016; Arora et al., 2017; Conneau et al., 2017; Cer et al., 2018; Zhang et al., 2018; Subramanian et al., 2018; Reimers & Gurevych, 2019; Zhang et al., 2020b).”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “…making it unclear which pretraining task is most effective or whether multiple pretraining tasks can be productively combined; in the related setting of sentence-to-vector encoding, multitask learning with multiple labeled datasets has yielded a robust state of the art (Subramanian et al., 2018).”
Mining Discourse Markers for Unsupervised Sentence Representation Learning (2019, methodology): “Besides, MTL is not a direct competitor to our approach since its main contribution is its multi-task setup, and it could benefit from using our training examples.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, methodology): “Recent works in sentence embeddings (Subramanian et al. 2018; Jernite, Bowman, and Sontag 2017) indicate that complementary aspects of the sentence (e.g. syntax, sentence length, word order) should be encoded in order for the model to produce sentence embeddings that are able to generalize over a…”