Continual Deep Learning by Functional Regularisation of Memorable Past

By Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E. Turner, Mohammad Emtiyaz Khan. 2020

174 citations recorded by the source corpus.

Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Meanwhile, sequential Bayesian inference over function space can be seen as a form of function regularization, which generally requires storing some old training samples (called “coreset” in literature), such as FRCL [127], FROMP [128] and S-FSVI [60].”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “As in related work (Lopez-Paz and Ranzato, 2017; Pan et al., 2020), we compute the forward transfer (FT) and backward transfer (BT) for S-FSVI on split CIFAR.”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Our experiments involve Reserovir sampling (RS) (Vitter, 1985), weighted reservoir sampling with the output-space Hessian (WRS), class-balanced reservoir sampling (CBRS) (Chrysakis & Moens, 2020), greedy gradient sample selection (GSS) (Aljundi et al., 2019b), feature-space clustering (FSS-Clust) (A”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [43,48,56,66,67,80,95] play regularization on model parameters to improve its stability.”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “While we enforce this stringent condition in this paper, our approach may be easily combined with memory-based techniques such as coresets or generative replay [8, 10, 13, 31, 33, 35, 37, 41, 42, 44, 47].”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “Alternatively, previous task functions can be regularized to ensure that the functions mapping inputs to outputs are remembered [38, 60, 49].”
Generalized Variational Continual Learning (2020, methodology): “In order to compare continual learning performance, we compare final average accuracy, forward transfer (the improvement on the current task as number of past tasks increases (Pan et al., 2020)) and backward transfer (the difference in accuracy between when a task is first trained and its accuracy a”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “explored in [58], however the usefulness of these approaches for class-IL is still to be determined.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “Variants of such approaches address catastrophic forgetting, and are useful for analyzing data instances (Pan et al., 2020; Krymolowski, 2002).”
Three types of incremental learning (2022, unknown): “For functional regularization, compared were learning without forgetting (LwF 50 ), which uses the inputs from the current context as anchor points; and functional regularization of the memorable past (FROMP 51 ), which uses stored examples from past contexts as anchor points.”
Formalizing the Generalization-Forgetting Trade-off in Continual Learning (2021, background): “For example, common solutions to the catastrophic forgetting issue include (1) representation-driven approaches [49, 25], (2) regularization approaches [27, 2, 34, 16, 48, 48, 24, 37, 10, 43], and (3) memory/experience replay [31, 32, 11, 12, 17].”
Hierarchical Indian buffet neural networks for Bayesian continual learning (2019, methodology): “3) and VCL has been shown to produce poor results when scaling to these larger models [Pan et al., 2020].”
Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping (2021, methodology): “The experimental implementation for most comparative methods mentioned in Table 3 have been taken from official implementations of [45], [50], [16] and [40].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Other methods convert the learned network into a Gaussian process (GP) and store a subset of samples from previous tasks in memory (Titsias et al., 2020; Pan et al., 2020).”
Provable Continual Learning via Sketched Jacobian Approximations (2021, background): “Subsequent works incorporated quadratic penalties on the weights that take interactions between the weights into account, and/or provided generalizations of EWC (Ritter et al., 2018; Liu et al., 2018; Chaudhry et al., 2018; Schwarz et al., 2018; Pan et al., 2021).”