EEC: Learning to Encode and Regenerate Images for Continual Learning

By Ali Ayub, Alan R. Wagner. 2021

63 citations recorded by the source corpus.

The two main impediments to continual learning are catastrophic forgetting and memory limitations on the storage of data. To cope with these challenges, we propose a novel, cognitively-inspired approach which trains autoencoders with Neural Style Transfer to encode and store images. During training on a new task, reconstructed images from encoded episodes are replayed in order to avoid catastrophic forgetting. The loss function for the reconstructed images is weighted to reduce its effect during classifier training to cope with image degradation. When the system runs out of memory the encoded episodes are converted into centroids and covariance matrices, which are used to generate pseudo-images during classifier training, keeping classifier performance stable while using less memory. Our approach increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets, while requiring 78% less storage space.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In contrast, autoencoder-based strategies, such as FearNet [210], SRM [361], CLEER [366], EEC [26], GMR [334] and Flashcards [138], can explicitly control the labels of the generated data, although relatively blurred.”
Class Gradient Projection For Continual Learning (2022, background): “Prevalent approaches to address the catastrophic forgetting problem in continual learning can be grouped into three broad categories: regularization-based approaches [2, 22, 30, 45], expansion-based approaches [17, 25, 31, 34, 36, 40], and memory-based approaches [3, 27, 33, 38].”
Few-Shot Continual Active Learning by a Robot (2022, background): “In recent years, several works in CL have focused on mitigating the catastrophic forgetting problem [13, 3, 6].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Ayub and Wagner [210] proposed a memory efficient method named Encoding Episodes as Concepts (EEC).”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “We kindly acknowledge that recent works which replay from a generator (close to our setting) also use small variants of ImageNet in their experiments [1, 3, 8].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Ayub and Wagner (2021) developed another unsupervised learning method based on neural style transformers (Gatys et al., 2016).”
Learning Novel Objects Continually Through Curiosity (2021, background): “To overcome catastrophic forgetting, many different deep learning techniques have been proposed for continual learning [12], [13], [14], [15], [16], [17].”
DDGR: Continual Learning with Deep Diffusion-based Generative Replay (2023, background): “Partial replay stores either all or a subset of previously learned inputs in a replay buffer, and mixes these inputs with new samples to train the classifier (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Rolnick et al., 2019; Ayub & Wagner, 2021).”
Tell me what this is: Few-Shot Incremental Object Learning by a Robot (2020, background): “To avoid storage of real samples, some approaches use generative-memory and regenerate samples of old classes using GANs or autoencoders [9], [10], [11], however the performance of these approaches is generally inferior to approaches that store real images.”
F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object Learning (2021, background): “Another genre of incremental learning approaches use generative memory such as autoencoders or GANs (Generative Adversarial Networks) [19], [6], [20].”
Open-World Continual Learning: Unifying Novelty Detection and Continual Learning (2023, methodology): “Generative methods construct a generative network to generate raw training data [56, 44, 4].”