An Empirical Investigation of the Role of Pre-training in Lifelong Learning

By Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, Emma Strubell. 2021

180 citations recorded by the source corpus.

The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due to its resemblance to biological learning but also its potential to reduce energy waste by obviating excessive model re-training. A key challenge to this paradigm is the phenomenon of catastrophic forgetting. With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, specifically with respect to catastrophic forgetting? We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classification tasks, including a large-scale study using a novel data set of 15 diverse NLP tasks. Across all settings, we observe that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initialized models. We then further investigate why pre-training alleviates forgetting in this setting. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading to wider minima. Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness to explicitly encourage wider basins during sequential fine-tuning. We show that this optimization approach outperforms several state-of-the-art task-sequential continual learning algorithms across multiple settings, occasionally even without retaining a memory that scales in size with the number of tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In addition to an earlier work on obtaining sparse representations through meta-training [185], recent works have attempted to incorporate the advantages of selfsupervised learning (SSL) [125], [281], [335] and large-scale pre-training [295], [380], [456] to improve the representations in initializa”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, background): “Such scenario is important for a twofold reason: i) as shown in [53], pre-training implicitly mitigates forgetting by widening the local minima found in function space, thus making the model more robust to input perturbations; additionally, ii) we accommodate for real-world scenarios where pre-train”
Continual Training of Language Models for Few-Shot Learning (2022, background): “…(Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classiﬁcation (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020; Mehta et al., 2021; Madotto et al., 2020).”
Transfer without Forgetting (2022, background): “Previous work underlined that naive pretraining is beneficial as it leads the learner to reduced forgetting [40].”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, background): “A similar study on pretraining for CL has been conducted by Mehta et al. [15].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “We chose a medium and a large buffer size for these rehearsal-based methods, based on recommendation of prior work [3, 39, 4, 60].”
Wide Neural Networks Forget Less Catastrophically (2021, background): “On the other hand, Mehta et al. (2021) showed that pre-trained over-parameterized networks forget considerably less than randomly initialized networks when trained continually.”
Continual Few-shot Intent Detection (2022, methodology): “At time step t , following (Mehta et al., 2021), we employ the average accuracy A t , average forgetting F t and learning accuracy LA t metrics after learning on the t -th task.”
Learning from One Continuous Video Stream (2023, background): “Recent continual learning papers reported that some of the challenges of the problem are partially mitigated by using features pretrained in IID settings [27, 36, 45].”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “More recently, Mehta et al. (2022) remarked that pre-training, for both NLP and CV, done on a diverse dataset signiﬁcantly reduced forgetting in downstream CL tasks.”
Fine-tuned Language Models are Continual Learners (2022, background): “This confirms contemporaneous findings by Cossu et al. (2022) and Mehta et al. (2021) in other setups and even modalities.”
Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Continual learning [2, 6, 26, 37, 39] aims to equip deep models with the capacity to continuously acquire new knowledge, e.g., learn to recognize new object categories, while handling catastrophic forgetting [18, 27, 28, 32] of the old knowledge.”