Online Learned Continual Compression with Adaptive Quantization Modules

By Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Joelle Pineau. 2019

97 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “To improve storage efﬁciency , AQM [136] performs online continual compression based on a VQ-VAE framework [137] and saves compressed data for replay.”
Memory Replay with Data Compression for Continual Learning (2022, positive): “One solution is to continually learn a generative model to replay generated data (Shin et al., 2017; Wu et al., 2018) or compress old training data (Caccia et al., 2020).”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, background): “Similarly, [36, 37] seek to be memory-efficient by saving feature vectors of instances from previous tasks, while learning a transformation from the feature space of past tasks to current ones.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…in some recent works that replay high-level feature representations (Hayes, Kafle, Shrestha, Acharya, & Kanan, 2020; Iscen, Zhang, Lazebnik, & Schmid, 2020; Caccia, Belilovsky, Caccia, & Pineau, 2020; Pellegrini, Graffieti, Lomonaco, & Maltoni, 2019; van de Ven, Siegelmann, & Tolias, 2020).”
Regularization Shortcomings for Continual Learning (2019, background): “In continual learning, we distinguish three types of memorization mechanisms: (1) dynamic architecture [25,18] where neural network use specific weights for specific tasks and can dynamically create new weights for new tasks, (2) replay methods where data from past tasks are saved to be replayed lat”
CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions (2020, methodology): “Then, they simply takes an SGD step with the concatenated samples, followed by an update of the memory (Chaudhry et al., 2019; Caccia et al., 2019).”
SPeCiaL: Self-Supervised Pretraining for Continual Learning (2021, methodology): “Instead, we assume that at each timestep t , input samples are drawn according to the conditional distribution P ( X t | C t ) , where C t ∈ C is a hidden time-dependent context variable [ Caccia et al. , 2020b ] , de-noting the state of the distribution at time t .”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, background): “Also related to our work, ExStream [29] clusters in a streaming fashion the training samples before pushing them into the replay buffer while [30] uses discrete autoencoders to compress the input data for rehearsing.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Prior work has exploited this intuition by compressing data in memory via a multilevel VAE that iteratively compresses samples to meet a fixed storage capacity (Caccia et al., 2020).”
CGLB: Benchmark Tasks for Continual Graph Learning (2022, methodology): “Better designs of memory replay are also explored recently [6, 9] since it is simple yet effective.”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “Others have explored sophisticated strategies for experience replay in DQN (e.g. Isele & Cosgun, 2018; Riemer et al., 2019; Wang et al., 2021; Caccia et al., 2020; Zhang & Sutton, 2017; Schaul et al., 2015) in lifelong RL, but these are not generative in nature.”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, methodology): “Latent replay: Consistency regularization in rehearsal-based CL can also be connected to generative-rehearsal based methods (van de Ven et al., 2020; Caccia et al., 2020) with latent representation replay.”
Class-Incremental Learning with Generative Classifiers (2021, background): “Important questions when storing data are which samples to store [37, 39] and in what format [9, 18].”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, unknown): “Numerous rehearsal-based methods have been developed to increase ER’s performance or efﬁciency (Aljundi et al., 2019a; Caccia et al., 2019; Hayes et al., 2018).”
Building a Subspace of Policies for Scalable Continual Learning (2022, methodology): “Other methods maintain a buffer of experience from prior tasks to alleviate forgetting (Lopez-Paz & Ranzato, 2017; Isele & Cosgun, 2018; Riemer et al., 2019b; Rolnick et al., 2019; Caccia et al., 2020a).”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “On the other hand, Caccia et al. (2019) proposed a comprehensive framework to characterize the various learning paradigm such as continual learning, meta-learning, continual meta-learning, and meta-continual learning.”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, background): “Continual learning research field studied many strategies, such as rehearsal [26, 9, 2, 3, 52, 19, 6], generative replay [46, 53, 28, 30, 40], dynamic architecture [16, 42, 32] or regularization [22, 55, 41, 44].”