Incremental Learning via Rate Reduction

By Ziyang Wu, Christina Baek, Chong You, Yi Ma. 2020

51 citations recorded by the source corpus.

Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes. The fundamental roadblock faced by deep learning methods is that the models are optimized as "black boxes," making it difficult to properly adjust the model parameters to preserve knowledge about previously seen data. To overcome the problem of catastrophic forgetting, we propose utilizing an alternative "white box" architecture derived from the principle of rate reduction, where each layer of the network is explicitly computed without back propagation. Under this paradigm, we demonstrate that, given a pretrained network and new data classes, our approach can provably construct a new network that emulates joint training with all past and new classes. Finally, our experiments show that our proposed learning algorithm observes significantly less decay in classification performance, outperforming state of the art methods on MNIST and CIFAR-10 by a large margin and justifying the use of "white box" algorithms for incremental learning even for sufficiently complex image data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

In Statistical Learning Theory

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For a regular network, the task-speciﬁc components could be parallel branches [233], [234], adaptive layers [58], [235], masks of intermediate features [186], [236], [237].”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Incremental Learning [22, 36, 48, 53, 58] continuously updates the model using data coming in a sequence of phases.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “In [39], instances are synthesized by a generative model [16] for rehearsal [30] test various exemplar selection strategies, showing that different ways of exemplar selection have a significant impact on performance and herding surpass other strategies in most settings.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “[211] proposed ReduNet which adapts the newly proposed ”white box" DNN derived from the rate reduction principle [212] to mitigate catastrophic forgetting in continual image classification.”
Co-Transport for Class-Incremental Learning (2021, background): “Class-incremental learning (CIL) is now a popular topic in the machine learning community [6, 9, 15, 25, 39, 47, 62, 68, 70].”
Incremental Learning via Rate Reduction (2021, background): “The work in this chapter was published at the Conference on Computer Vision and Pattern Recognition (CVPR) 2021 by the author, Ziyang Wu, Chong You, and Yi Ma [23].”