Kernel Continual Learning

By Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao, Cees G. M. Snoek. 2021

48 citations recorded by the source corpus.

This paper introduces kernel continual learning, a simple but effective variant of continual learning that leverages the non-parametric nature of kernel methods to tackle catastrophic forgetting. We deploy an episodic memory unit that stores a subset of samples for each task to learn task-specific classifiers based on kernel ridge regression. This does not require memory replay and systematically avoids task interference in the classifiers. We further introduce variational random features to learn a data-driven kernel for each task. To do so, we formulate kernel continual learning as a variational inference problem, where a random Fourier basis is incorporated as the latent variable. The variational posterior distribution over the random Fourier basis is inferred from the coreset of each task. In this way, we are able to generate more informative kernels specific to each task, and, more importantly, the coreset size can be reduced to achieve more compact memory, resulting in more efficient continual learning based on episodic memory. Extensive evaluation on four benchmarks demonstrates the effectiveness and promise of kernels for continual learning.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “The second is online variational inference (VI) [7], [90], [204], [233], [245], [276], [314], [369], [400].”
Architecture Matters in Continual Learning (2022, background): “…Riemer et al., 2018) or incorporate them as part of the optimization process (e.g., gradient projection) (Farajtabar et al., 2020; Chaudhry et al., 2018; Balaji et al., 2020), or learn a generative model (Shin et al., 2017; Kirichenko et al., 2021) or kernel (Derakhshani et al., 2021) to do so.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, methodology): “each comprises of 10 classes with 500 and 100 32 32 images each for training and testing respectively; Split miniImageNet [15], [47], [48] derives from miniImageNet [49], a 100-class subset of the popular ImageNet dataset, split in 20 classification tasks.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Derakhshani et al. [114] proposed Kernel Continual Learning, which uses the stored data instances to train a non-parametric classifier with kernel ridge regression.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “One nonparametric kernel method uses an episodic memory for detecting the task at inference time instead of for replay (Derakhshani et al., 2021).”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “In a general continual learning [7] the kernel was shown to be effective for regression.”