Continual Learning with Fully Probabilistic Models

By Benedikt Pfülb, Alexander Rainer Tassilo Gepperth, Benedikt Bagus. 2021

10 citations recorded by the source corpus.

We present an approach for continual learning (CL) that is based on fully probabilistic (or generative) models of machine learning. In contrast to, e.g., GANs that are"generative"in the sense that they can generate samples, fully probabilistic models aim at modeling the data distribution directly. Consequently, they provide functionalities that are highly relevant for continual learning, such as density estimation (outlier detection) and sample generation. As a concrete realization of generative continual learning, we propose Gaussian Mixture Replay (GMR). GMR is a pseudo-rehearsal approach using a Gaussian Mixture Model (GMM) instance for both generator and classifier functionalities. Relying on the MNIST, FashionMNIST and Devanagari benchmarks, we first demonstrate unsupervised task boundary detection by GMM density estimation, which we also use to reject untypical generated samples. In addition, we show that GMR is capable of class-conditional sampling in the way of a cGAN. Lastly, we verify that GMR, despite its simple structure, achieves state-of-the-art performance on common class-incremental learning problems at very competitive time and memory complexity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Tianyu Pang
6. Yinpeng Dong
7. Vamsi Krishna Ithapu
8. Mirella Lapata

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Yinpeng Dong
6. Vamsi Krishna Ithapu
7. Alan L. Yuille
8. Tianyu Pang

In Probabilistic & Bayesian Methods

1. Liyuan Wang
2. Jun Zhu
3. Hang Su
4. Xingxing Zhang

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Simone Scardapane
6. Valerio Marsocci
7. Liyuan Wang
8. Hongwei Yan

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In contrast, autoencoder-based strategies, such as FearNet [210], SRM [361], CLEER [366], EEC [26], GMR [334] and Flashcards [138], can explicitly control the labels of the generated data, although relatively blurred.”