Improving and Understanding Variational Continual Learning

By Siddharth Swaroop, Cuong V Nguyen, Thang D. Bui, Richard E. Turner. 2019

53 citations recorded by the source corpus.

In the continual learning setting, tasks are encountered sequentially. The goal is to learn whilst i) avoiding catastrophic forgetting, ii) efficiently using model capacity, and iii) employing forward and backward transfer learning. In this paper, we explore how the Variational Continual Learning (VCL) framework achieves these desiderata on two benchmarks in continual learning: split MNIST and permuted MNIST. We first report significantly improved results on what was already a competitive approach. The improvements are achieved by establishing a new best practice approach to mean-field variational Bayesian neural networks. We then look at the solutions in detail. This allows us to obtain an understanding of why VCL performs as it does, and we compare the solution to what an `ideal' continual learning solution might be.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “For instance, variational continual learning (VCL) [29, 45] applied a sequence of variational approximations to the true posterior and used the current approximate posterior as prior for the new task.”
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, neutral): “Additional results from [42, 45].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “The second is online variational inference (VI) [7], [90], [204], [233], [245], [276], [314], [369], [400].”
Continual Learning via Sequential Function-Space Variational Inference (2023, positive): “Variational continual learning ( VCL ; Nguyen et al., 2018; Swaroop et al., 2019), which extends the concept of online variational inference (Broderick et al., 2013; Ghahramani and Attias, 2000; Honkela and Valpola, 2003; Sato, 2001) to deep neural networks, uses the parameter covariance matrix of…”
Posterior Meta-Replay for Continual Learning (2021, methodology): “Variational continual learning (VCL, Nguyen et al. [53], Swaroop et al. [69]) was introduced as a way to continually learn a single mean-ﬁeld Gaussian posterior approximation across tasks by doing a recursive Bayesian update via VI.”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “Split MNIST results for SI (Zenke et al., 2017) and EWC (Kirkpatrick et al., 2017) are taken from VCL (Swaroop et al., 2019), however, are not directly comparable as they use a multi-head setup making evaluation easier; CS = coreset.”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “We also run FROMP on a smaller network architecture (Swaroop et al., 2019), obtaining 99 .”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “IUVCL [82] proposes a new best-practice approach to mean-ﬁeld variational Bayesian neural networks.”
Practical Deep Learning with Bayesian Principles (2019, methodology): “Additionally, as found in the batch learning setting, VOGN is much quicker than BBB: we run VOGN for only 100 epochs per task, whereas VCL requires 800 epochs per task to achieve best results [52].”
Indian Buffet Neural Networks for Continual Learning (2019, methodology): “Previous Bayesian approaches have involved Laplace approximations [1, 10, 7] and variational inference [2, 11, 4], to aid in computational tractability.”
Understanding Regularisation Methods for Continual Learning (2020, methodology): “Following [42], but unlike [44], we re-initialise model variables (fully connected layers, etc.”
Drinking From a Firehose: Continual Learning With Web-Scale Natural Language (2020, methodology): “leverage a data-driven prior from past tasks to regularize the current task [4], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25].”
Overcoming Catastrophic Forgetting by Bayesian Generative Regularization (2019, methodology): “In particular, we implement the improved version of VCL (Swaroop et al., 2019).”