Fine-tuned Language Models for Text Classification

By Jeremy Howard, Sebastian Ruder. 2018

284 citations recorded by the source corpus.

Transfer learning has revolutionized computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Fine-tuned Language Models (FitLaM), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a state-of-the-art language model. Our method significantly outperforms the state-of-the-art on five text classification tasks, reducing the error by 18-24% on the majority of datasets. We open-source our pretrained models and code to enable adoption by the community.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “Pretraining methods have been designed with different training objectives, including language modeling (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018), machine translation (McCann et al., 2017), and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “Recent parallel work (Ramachandran et al., 2016; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019) further builds upon these ideas by not just transferring the language model to extract…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “The embeddings [Collobert and Weston, 2008, Collobert et al., 2011], intermediate representations [Peters et al., 2018], and weights of language models can be transferred to similar architectures [Ramachandran et al., 2017] and classification tasks [Howard and Ruder, 2018].”
A Simple Method for Commonsense Reasoning (2018, background): “Neural LMs have also been applied successfully to improve downstream applications [18, 19, 20, 21].”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “The language models themselves have been transferred to new tasks through ﬁne-tuning as well (Dai & Le, 2015; Radford et al., 2018; Howard & Ruder, 2018).”
A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, methodology): “A common approach to transfer learning in NLP is to train a language model (LM) on large amounts of unsupervised text (Howard and Ruder, 2018) and use it, with or without further ﬁne-tu-ning, to solve other downstream tasks.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “, 2018), or when the full model is finetuned on transfer tasks (Radford et al.; Howard and Ruder, 2018).”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “Neural network models are now the state-of-the-art for a wide range of tasks such as text classiﬁ-cation (Howard and Ruder, 2018), machine translation (Vaswani et al., 2017), semantic role labeling (Zhou and Xu, 2015; He et al., 2017), coreference resolution (Lee et al., 2017a), and semantic…”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “…current dominant paradigm in natural language processing relies heavily on pre-trained models: large language models that can then be fine-tuned on a downstream task (Howard and Ruder, 2018; Devlin et al., 2018) or even used as-is without additional data (Radford et al., 2019; Brown et al., 2020).”
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, methodology): “Unsupervised and self-supervised pre-training methods, such as ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018), and more recently BERT (Devlin et al., 2018), GPT and GPT-2 (Radford et al., 2018, 2019) and XLNet (Yang et al., 2019) have established a qualitatively new level of baseline pe”
All You Need is "Love": Evading Hate Speech Detection (2018, methodology): “Recently, transfer learning has been argued to improve performance text classification [12].”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, background): “Transfer learning with language models Extending upon the success of pretrained word embeddings (Mikolov et al., 2013), contextual word vectors based on LSTMs first demonstrated strong results across discriminative NLU tasks (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018).”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “These include visual object recognition, speech recognition and production, and basic natural language prediction and comprehension [1, 2, 3, 4].”
A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media (2019, background): “The year 2018 has been an inﬂection point for diﬀer-ent NLP tasks thanks to remarkable breakthroughs: Universal Language Model Fine-Tuning (ULMFiT) [9], Embedding from Language Models (ELMO) [17], OpenAI s Generative Pre-trained Transformer (GPT) [18], and Googles BERT model [4].”