Class-Incremental Learning with Generative Classifiers

By Gido M. van de Ven, Zhe Li, Andreas S. Tolias. 2021

77 citations recorded by the source corpus.

Incrementally training deep neural networks to recognize new classes is a challenging problem. Most existing class-incremental learning methods store data or use generative replay, both of which have drawbacks, while ‘rehearsal-free’ alternatives such as parameter regularization or bias-correction methods do not consistently achieve high performance. Here, we put forward a new strategy for class-incremental learning: generative classification. Rather than directly learning the conditional distribution p(y|x), our proposal is to learn the joint distribution p(x, y), factorized as p(x|y)p(y), and to perform classification using Bayes’ rule. As a proof-of-principle, here we implement this strategy by training a variational autoencoder for each class to be learned and by using importance sampling to estimate the likelihoods p(x|y). This simple approach performs very well on a diverse set of continual learning benchmarks, outperforming generative replay and other existing baselines that do not store data.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, neutral): “Note that task-free and class-incremental are compatible (Jin et al., 2020; van de Ven et al., 2021).”
Divide and not forget: Ensemble of selectively trained experts in Continual Learning (2024, background): “In an extreme case, each task can have a dedicated expert network (Aljundi et al., 2017) or a single network per class (van de Ven et al., 2021).”
Class-Incremental Learning: A Survey (2023, background): “…[89], [110], [111] Weight Rectify [112], [113], [114] § 3.7: Template-Based Classification [37], [82], [104], [115], [116], [117], [118], [119], [120], [121], [122], [123] Note that the classification rules of these categories are based on the ‘key point’ (or the special focus) of each algorithm.”
Three types of incremental learning (2022, unknown): “55 , which uses class-specific generative models as templates.”