Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting

By Vinod Kumar Kurmi, Badri N. Patro, Venkatesh Subramanian, Vinay P. Namboodiri. 2021

26 citations recorded by the source corpus.

One of the major limitations of deep learning models is that they face catastrophic forgetting in an incremental learning scenario. There have been several approaches proposed to tackle the problem of incremental learning. Most of these methods are based on knowledge distillation and do not adequately utilize the information provided by older task models, such as uncertainty estimation in pre-dictions. The predictive uncertainty provides the distributional information can be applied to mitigate catastrophic forgetting in a deep learning framework. In the proposed work, we consider a Bayesian formulation to obtain the data and model uncertainties. We also incorporate self-attention framework to address the incremental learning problem. We define distillation losses in terms of aleatoric uncertainty and self-attention. In the proposed work, we investigate different ablation analyses on these losses. Furthermore, we are able to obtain better results in terms of accuracy on standard benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Jun Liu
4. Liyuan Wang
5. Xingxing Zhang
6. Hossein Rahmani
7. Li Xu
8. Bryan M. Williams

In Computer Vision and Pattern Recognition

1. Hossein Rahmani
2. Jun Liu
3. Hang Su
4. Jun Zhu
5. Liyuan Wang
6. Xingxing Zhang
7. Li Xu
8. Bryan M. Williams

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Jun Liu
6. Hossein Rahmani
7. Li Xu
8. Bryan M. Williams

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “To adequately exploit the past information, AU [234] incorporates uncertainty and self-attention into the distillation loss, while CSC [22] additionally leverages the structure of the feature space.”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “[54] utilized the prediction uncertainty of the model on the previous tasks to mitigate catastrophic forgetting.”