Energy-based Latent Aligner for Incremental Learning

By K. J. Joseph, Salman Hameed Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Vineeth Nallure Balasubramanian. 2022

57 citations recorded by the source corpus.

Deep learning models tend to forget their earlier knowledge while incrementally learning new tasks. This behavior emerges because the parameter updates optimized for the new tasks may not align well with the updates suitable for older tasks. The resulting latent representation mismatch causes forgetting. In this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which first learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values. This learned manifold is used to counter the representational shift that happens during incremental learning. The implicit regularization that is offered by our proposed methodology can be used as a plug-and-play module in existing incremental learning methodologies. We validate this through extensive evaluation on CIFAR-100, ImageNet subset, ImageNet1k and Pascal VOC datasets. We observe consistent improvement when ELI is added to three prominent methodologies in class-incremental learning, across multiple incremental settings. Further, when added to the state-of-the-art incremental object detector, ELI provides over 5% improvement in detection accuracy, corroborating its effectiveness and complementary advantage to the existing art. Code is available at: https://github.com/JosephKJ/ELI.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “ELI [151] learns an energy manifold with the old and new models to realign the representation shifts for optimizing incremental tasks.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “A recent method ELI [22], introduces an energy based implicit regularizer which helps to alleviate forgetting.”
Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning (2023, methodology): “…rely on evolving the prototypes (Zhang et al., 2021; Zhu et al., 2021a) or sophisticated designs of loss and regularizer (Ren et al., 2019; Hou et al., 2019; Tao et al., 2020a; Joseph et al., 2022; Lu et al., 2022; Hersche et al., 2022; Chen & Lee, 2021; Aky¨urek et al., 2022; Yang et al., 2022a).”
Continual Detection Transformer for Incremental Object Detection (2023, unknown): “Recent incremental learning methods can be divided into two categories: (i) Knowledge Distillation (KD) tries to preserve the knowledge capture in a previous version of the model by matching logits [26,41], feature maps [11], or other information [18, 39, 45, 48, 51] in the new model.”