Lifelong Generative Modelling Using Dynamic Expansion Graph Model
By Fei Ye, Adrian G. Bors. 2021
14 citations recorded by the source corpus.
Variational Autoencoders (VAEs) suffer from degenerated performance, when learning several successive tasks. This is caused by catastrophic forgetting. In order to address the knowledge loss, VAEs are using either Generative Replay (GR) mechanisms or Expanding Network Architectures (ENA). In this paper we study the forgetting behaviour of VAEs using a joint GR and ENA methodology, by deriving an upper bound on the negative marginal log-likelihood. This theoretical analysis provides new insights into how VAEs forget the previously learnt knowledge during lifelong learning. The analysis indicates the best performance achieved when considering model mixtures, under the ENA framework, where there are no restrictions on the number of components. However, an ENA-based approach may require an excessive number of parameters. This motivates us to propose a novel Dynamic Expansion Graph Model (DEGM). DEGM expands its architecture, according to the novelty associated with each new database, when compared to the information already learnt by the network from previous tasks. DEGM training optimizes knowledge structuring, characterizing the joint probabilistic representations corresponding to the past and more recently learned tasks. We demonstrate that DEGM guarantees optimal performance for each task while also minimizing the required number of parameters.
Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Cancer Research
In Computer Vision and Pattern Recognition
Citations in the corpus
- Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “One of the most popular approaches to relieve forgetting is by imposing a regularization loss within the optimization procedure [7, 11, 13, 16, 19, 25, 26, 31, 34, 35, 40, 41, 56], where the network’s parameters deemed important to the past tasks are penalized when updating.”
- Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “This issue can be solved by storing the information of past samples into the network’s parameters which are then frozen when learning novel tasks [35,64,65,67,68].”
- Learning an Evolved Mixture Model for Task-Free Continual Learning (2022, background): “Another approach for TFCL is based on the expansion of the network architecture or by adding new components to a mixture model [19, 20, 21].”