Universal Language Model Fine-tuning for Text Classification

By Jeremy Howard, Sebastian Ruder. 2018

4,208 citations recorded by the source corpus.

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100 times more data. We open-source our pretrained models and code.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “A recent trend in transfer learning from language models (LMs) is to pre-train some model architecture on a LM objective before fine-tuning that same model for a supervised downstream task (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018).”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “…noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pen-nington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Rad-ford et al., 2018).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “, 2018), text classification (Dai and Le, 2015; Howard and Ruder, 2018), sequence tagging (Peters et al.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Early results on transfer learning for NLP leveraged recurrent neural networks [Peters et al., 2018; Howard and Ruder, 2018], but it has recently become more common to use models based on the “Transformer” architecture [Vaswani et al., 2017].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “Pretraining methods have been designed with different training objectives, including language modeling (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018), machine translation (McCann et al., 2017), and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “This network is then fine-tuned (performs transfer learning) on the benchmark’s supervised tasks, each of which include a much smaller number of training examples than the pretraining corpus (Howard and Ruder, 2018; Peters et al., 2018).”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, neutral): “, 2014) and pretrained language models (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “In the past few years, it has become increasingly common to pre-train a language model using an unsupervised objective on a large, unstructured text corpus before ﬁne-tuning it on a downstream task of interest (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Providing hints in the form of natural language patterns for zero-shot learning was first proposed by Radford et al. (2019), who used this idea for challenging tasks such as reading comprehension, machine translation and question answering.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Motivated by the recent success in self-supervised language representation learning (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Song et al., 2019; Zhang et al., 2019; Yang et al., 2019), we propose several strategies that pre-train the encoder and…”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, neutral): “Full network pre-training (Dai & Le, 2015; Radford et al., 2018; Devlin et al., 2019; Howard & Ruder, 2018) has led to a series of breakthroughs in language representation learning.”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “This method often substantially outperforms training on the supervised datasets from scratch, and a single pretrained language model often can be ﬁne-tuned for state of the art performance on many different supervised datasets (Howard and Ruder, 2018).”
Release Strategies and the Social Impacts of Language Models (2019, background): “Other language models have accelerated NLP research and applications by providing better starting points for supervised training models [14], introducing techniques for fine-tuning [32], and enhancing performance in challenges like question answering and sentiment analysis [52].”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “Fine-tuning approaches mainly pre-train a model on language modeling objective and then ﬁne-tune the model on the downstream tasks with supervised data (Dai & Le, 2015; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018).”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “The latter ﬁnetuning strategy is motivated by the fact that ELMo [49] and ULMFiT [26] show that ﬁne-tuning language models on domain speciﬁc data can lead to performance improvements.”
Cross-lingual Language Model Pretraining (2019, methodology): “Generative pretraining of sentence encoders (Rad-ford et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018) has led to strong improvements on numerous natural language understanding benchmarks (Wang et al., 2018).”