An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

By Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos. 2019

143 citations recorded by the source corpus.

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “With LCE denoting cross-entropy loss and LMLM denoting language modeling loss, we compute the final loss as L = (1− α) · LCE + α · LMLM This idea was recently applied by Chronopoulou et al. (2019), who employ auxiliary language modeling in a data-rich scenario.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “Of course, there are further ways to leverage unlabeled data such as keeping an auxiliary language modeling objective during ﬁnetuning (Chronopoulou et al., 2019).”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “Related approaches include language modeling as an auxiliary objective to task classifier fine-tuning (Chronopoulou et al., 2019; Radford et al., 2018) or consider simple syntactic structure of the input while adapting to task-specific data (Swayamdipta et al.”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “The catastrophic forgetting issue in NLP tasks has raised increasing attention recently (Mou et al., 2016; Chronopoulou et al., 2019).”
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (2019, background): “Interest in learning general-purpose representations for natural language through unsupervised, multi-task and transfer learning has been sky-rocketing lately (Peters et al., 2018; Radford et al., 2018; McCann et al., 2018; Chronopoulou et al., 2019; Phang et al., 2018; Wang et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, methodology): “[21] proposed a simpler unfreezing method, sequential unfreezing, which first fine-tunes only the randomly-initialized task-specific layers, and then unfreezes the hidden layers of PTM, and finally unfreezes the embedding layer.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, background): “…models have achieved success in improving systems in language understanding, leading to state-of-the-art results on a wide range of tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; McCann et al., 2017; Yang et al., 2019; Chronopoulou et al., 2019; Lample and Conneau, 2019).”
Putting Machine Translation in Context with the Noisy Channel Model (2019, background): “…have achieved great success in improving systems in language understanding, leading to state-of-the-art results on a wide range of tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; McCann et al., 2017; Yang et al., 2019; Chronopoulou et al., 2019; Lample & Conneau, 2019).”
Generating Abstractive Summaries with Finetuned Language Models (2019, methodology): “…language models based on the neural Trans-former architecture (Vaswani et al., 2017) have shown promising results in language understanding tasks (Houlsby et al., 2019; Devlin et al., 2018; Chronopoulou et al., 2019), but so far have had limited success in generation tasks (Zhang et al., 2019).”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, background): “Transfer learning with a single-step auxiliary loss [21] is yet another transfer learning approach that adopts multi-task learning perspective via the addition of a pretraining objective in the finetuning stage.”
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, unknown): “…in terms of language resource availability (Joshi et al., 2020), the advent of Language Models (LMs) has indisputably enabled a myriad of multilingual architectures to ﬂuorish, ranging from LSTMs to the arguably more popular transformer-based models (Chronopoulou et al., 2019; Pires et al., 2019).”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “Catastrophic forgetting in NLP has raised increased attention recently (Mou et al., 2016; Arora et al., 2019; Chronopoulou et al., 2019).”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, methodology): “Chronopoulou et al. (2019) use an auxiliary LM loss to avoid catastrophic forgetting in transfer learning and Jia et al. (2019) adopt this approach for cross-domain named-entity recognition.”
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, background): “Many endeavors have been made to discuss the scaling laws for pre-training (Anil et al., 2023; Hoffmann et al., 2022), transfer learning (Chronopoulou et al., 2019), preference modeling (Gao et al., 2022) and mathematical reasoning (Yuan et al., 2023b).”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, methodology): “Chronopoulou et al. (2019) and Radford et al. (2018) propose to train task-speciﬁc parameters for PLMs by using an auxiliary LM loss on target domains.”
Bridging the domain gap in cross-lingual document classification (2019, methodology): “(2018); Chronopoulou et al. (2019)). It is natural to use unsupervised domain data for this task, when available (Lee et al. (2019); Han & Eisenstein (2019)). The study of weakly supervised learning in language processing is relatively new (Johnson & Zhang, 2016; Yu et al., 2018). Most recently, Xie”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, background): “ and then ﬁnetune the language model and task-speciﬁc head on supervised data. Optimizing the effectiveness of this approach has been the focus of much study (Houlsby et al., 2019; Wang et al., 2019; Chronopoulou et al., 2019). Given the success of pretraining for NLU tasks, how can large language m”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “, 2020) and auxiliary language modeling (Chronopoulou et al., 2019), both of which were shown to be helpful by Schick and Schütze, (2020).”