Meta-Consolidation for Continual Learning

By K. J. Joseph, Vineeth Nallure Balasubramanian. 2020

63 citations recorded by the source corpus.

The ability to continuously learn and adapt itself to new tasks, without losing grasp of already acquired knowledge is a hallmark of biological learning systems, which current deep learning systems fall short of. In this work, we present a novel methodology for continual learning called MERLIN: Meta-Consolidation for Continual Learning. We assume that weights of a neural network $\boldsymbol \psi$, for solving task $\boldsymbol t$, come from a meta-distribution $p(\boldsymbol{\psi|t})$. This meta-distribution is learned and consolidated incrementally. We operate in the challenging online continual learning setting, where a data point is seen by the model only once. Our experiments with continual learning benchmarks of MNIST, CIFAR-10, CIFAR-100 and Mini-ImageNet datasets show consistent improvement over five baselines, including a recent state-of-the-art, corroborating the promise of MERLIN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “To address this issue, the i th task’s distribution p ( D t | i, θ ) could be recovered by replay [74], [78], [79], [80] p ( where the marginal task distribution p ( i ) ∝ N i in general.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Meta-learning based methods [21, 26] are also useful while learning continually.”
Novel Class Discovery without Forgetting (2022, background): “Incremental learning methods [41,11,27] learn a single model across tasks, but data for each incremental task is fully annotated.”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “…classi-ﬁers, we experiment with two prominent protocols that exist in the literature: a) train with half the total number of classes as the ﬁrst task [20,31], and equal number of classes per task thereafter, b) ensure that each task (including the ﬁrst) has equal number of classes [7, 24, 38, 41].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Joseph and Balasubramanian [124] recently proposed to incorporate the idea of meta learning in optimization.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, background): “that these classic settings lack realism [12, 13] and consequently define new scenarios by imposing restrictions on what models are allowed to do while learning.”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, methodology): “It has been used as a training strategy in sequence learning [43, 44] and continual learning scenarios [45, 46, 47, 48].”
Continual Learning with Fully Probabilistic Models (2021, background): “Training and Evaluation Paradigms for CL In the context of CL, a wide range of training and evaluation paradigms are proposed, see [27, 28, 29, 30, 5, 31, 32].”
Towards Open World Object Detection (2021, methodology): “We note that a number of involved approaches have been developed to alleviate such forgetting, including methods based on parameter regularization [2, 24, 29, 66], exemplar replay [6, 51, 37, 5], dynamically expanding networks [39, 60, 56] and meta-learning [50, 25].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “La-MAML [65] proposes an efficient gradient-based meta-learning that incorporates per-parameter learning rates for online CL. MERLIN [67] proposes an online CL method based on consolidation in a meta-space, namely, the latent space that generates model weights for solving downstream tasks.”
Incremental Object Detection via Meta-Learning (2020, neutral): “In comparison to the incremental setting for image-classification [14], [15], [16], [17], class-incremental object detection and class-incremental semantic segmentation [18], [19] has been relatively less explored.”
Hypernetworks for Continual Semi-Supervised Learning (2021, unknown): “Further, L ( θ , φ | Θ , t ) can be expressed as (refer to [ Joseph and Balasubramanian, 2020 ] for complete derivation): [log p θ ( Θ | z , t )] (5) Maximizing Eqn.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Joseph and Balasubramanian (2020) used regularization at a meta level, learning a generative regularized hypernetwork using a variational autoencoder (VAE) to generate parameters for each task based on a task descriptor.”
Continual Learning with Deep Artificial Neurons (2020, background): “umption. Broadly speaking, 4 categories of techniques have emerged to combat catastrophic forgetting: (1) regularization-based approaches [4, 22, 24], (2) knowledge-compression, or capacity expansion [20, 23, 28, 38], (3) experience replay and/or external memory slots [9, 29, 36, 40], and (4) metale”