Energy-Based Models for Continual Learning

By Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, Igor Mordatch. 2020

48 citations recorded by the source corpus.

We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems. Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support a dynamically-growing number of tasks or classes that causes less interference with previously learned information. We find that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks. We also show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks. These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Energy-based Latent Aligner for Incremental Learning (2022, methodology): “A recent unpublished work [27] proposes to replace the standard softmax layer of an incremental model with an energy-based classiﬁer head.”
Posterior Meta-Replay for Continual Learning (2021, methodology): “Results obtained with a Resnet-32 [18] (Table 3) show performance gains in the task-agnostic setting compared to recent methods like EBM [42], and to the PriorFocused method VCL.”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “Recently, several works for class-IL started addressing this setting [117], [120], [121].”
Class-Incremental Learning: A Survey (2023, methodology): “Finally, we can use an energy-based model [205] to compute for each class an energy value rather than a likelihood [206] to alleviate the high cost of the generative model.”
Class-Incremental Learning with Generative Classifiers (2021, methodology): “When a standard softmax-based classifier is trained on a class-incremental learning problem, it ends up predicting only the most recently seen classes [29].”
Mitigating Forgetting in Online Continual Learning with Neuron Calibration (2021, background): “Though the research community has also started to pay attention on a number of non replay-based directions to tackle the continual learning problems, such as leveraging energy-based models [14] and neuron modulation approaches [15], the replay-based methods still play a pivot role in continual learn”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “, 2019), continual learning (Li et al., 2020), protein design and folding (Ingraham et al.”
Three types of incremental learning (2022, unknown): “An advantage of distinguishing between the context set and the data stream is that it makes it possible to describe continual learning problems with gradual transitions between contexts 34,37,40,42 or whereby contexts are revisited 43–45 .”