Semi-supervised Sequence Learning

By Andrew M. Dai, Quoc V. Le. 2015

1,296 citations recorded by the source corpus.

We present two approaches to use unlabeled data to improve Sequence Learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a language model in NLP. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" algorithm for a later supervised sequence learning algorithm. In other words, the parameters obtained from the pretraining step can then be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better. With pretraining, we were able to achieve strong performance in many classification tasks, such as text classification with IMDB, DBpedia or image recognition in CIFAR-10.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “A recent trend in transfer learning from language models (LMs) is to pre-train some model architecture on a LM objective before fine-tuning that same model for a supervised downstream task (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018).”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “…noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pen-nington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Rad-ford et al., 2018).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “…for many NLP tasks including question answering, coreference resolution, and semantic role labeling (Peters et al., 2018), text classiﬁcation (Dai and Le, 2015; Howard and Ruder, 2018), sequence tagging (Peters et al., 2017), sequence–to–sequence learning (Ramachandran et al., 2017), and…”
Deep Contextualized Word Representations (2018, background): “Dai and Le (2015) and Ramachandran et al. (2017) pretrain encoder-decoder pairs using language models and sequence autoencoders and then ﬁne tune with task speciﬁc supervision.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Dai and Le (2015) ﬁrst proposed ﬁne-tuning a language model (LM) but require millions of in-domain documents to achieve good performance, which severely limits its applicability.”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et…”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “However, recent work has shown some success in using unsupervised pretraining with unlabeled data to both improve the performance of these methods and extend them to lower-resource settings (Dai and Le, 2015; Kiros et al., 2015; Ba-jgar et al., 2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “Dai and Le (2015) pre-trained LSTMs using language models and sequence autoencoders then ﬁne tuned the weights for classiﬁcation tasks.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “Pretraining methods have been designed with different training objectives, including language modeling (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018), machine translation (McCann et al., 2017), and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “The resulting pre-trained model can then be further trained (fine-tuned) for a downstream task of primary interest (in our case, OpenQA), often leading to better generalization than training from scratch (Dai & Le, 2015; Radford et al., 2019).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “…with Language Models In the past few years, it has become increasingly common to pre-train a language model using an unsupervised objective on a large, unstructured text corpus before ﬁne-tuning it on a downstream task of interest (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018).”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “…the shift from pre-training word embeddings, whether standard (Mikolov et al., 2013; Pennington et al., 2014) or contextualized (McCann et al., 2017; Peters et al., 2018), to full-network pre-training followed by task-specific fine-tuning (Dai & Le, 2015; Radford et al., 2018; Devlin et al., 2019).”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “One successful method has been to pretrain a large generative language model on a corpus of unsupervised data, then fine-tune the model for supervised NLP tasks (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Khandelwal et al., 2019). This method often substantially outperforms trainin”
Release Strategies and the Social Impacts of Language Models (2019, background): “Other language models have accelerated NLP research and applications by providing better starting points for supervised training models [14], introducing techniques for fine-tuning [32], and enhancing performance in challenges like question answering and sentiment analysis [52].”