Reinforced Continual Learning

By Ju Xu, Zhanxing Zhu. 2018

433 citations recorded by the source corpus.

Most artificial intelligence models have limiting ability to solve new tasks faster, without forgetting previously acquired knowledge. The recently emerging paradigm of continual learning aims to solve this issue, in which the model learns various tasks in a sequential fashion. In this work, a novel approach for continual learning is proposed, which searches for the best neural architecture for each coming task via sophisticatedly designed reinforcement learning strategies. We name it as Reinforced Continual Learning. Our method not only has good performance on preventing catastrophic forgetting but also fits new tasks well. The experiments on sequential classification tasks for variants of MNIST and CIFAR-100 datasets demonstrate that the proposed approach outperforms existing continual learning alternatives for deep networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The dynamic architecture can be explicitly optimized to improve parameter efﬁciency and knowledge transfer, such as by reinforcement learning [229], [230], architecture search [230], [231], variational Bayes [232], etc.”
Overcoming Multi-Model Forgetting (2019, neutral): “In (Xu & Zhu, 2018), forgetting is avoided altogether by ﬁxing the parameters of the ﬁrst model while complementing the second one with additional operations found by an architecture search procedure.”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “Of the methods that grow the amount of neural resources, some utilise the deﬁnition of task boundaries (Rusu et al., 2016; Xu & Zhu, 2018), while others do not (Ring, 1997; Zhou et al., 2012), but since the computational complexity grows with the size of the network, it is not clear that they are…”
Learnability and Algorithm for Continual Learning (2023, methodology): “Many existing approaches have used similar approaches (Ritter et al., 2018; Schwarz et al., 2018; Xu & Zhu, 2018; Castro et al., 2018; Dhar et al., 2019; Lee et al., 2019; Ahn et al., 2019; Liu et al., 2020a).”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “In contrast, the related work [14, 54] could use only current-phase data to estimate a short-term reward.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Many other papers use related approaches (Zenke, Poole, and Ganguli 2017; Fernando et al. 2017; Aljundi et al. 2017; Ritter, Botev, and Barber 2018; Xu and Zhu 2018; Kemker and Kanan 2018; Parisi et al. 2018; Ahn et al. 2019; Hu et al. 2021; Dhar et al. 2019; Adel, Zhao, and Turner 2020).”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Other non-exemplar-based methods change the network structure by expanding and pruning to meet the requirements of new tasks [31], [32], [33], [34].”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “…ranging from the dynamic architectures (Rusu et al., 2016; Serra et al., 2018; von Oswald et al., 2020; Yoon et al., 2018; Li et al., 2019; Xu & Zhu, 2018), experience replay (Lin, 1992; Chaudhry et al., 2019b; Aljundi et al., 2019a; Riemer et al., 2019; Rolnick et al., 2019; Wu et al.,…”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, neutral): “In order to ﬁnd the optimal structure for each of the sequential tasks, Reinforced Continual Learning (RCL) (Xu & Zhu, 2018) leverages reinforcement learning and (Li et al., 2019) adapts architecture search.”
Architecture Matters in Continual Learning (2022, background): “There have been some eﬀorts on applying architecture search to continual learning (Xu and Zhu, 2018; Huang et al., 2019; Gao et al., 2020; Li et al., 2019; Lee et al., 2021).”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Expansion-based methods (Rusu et al., 2016; Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu and Zhu, 2018; Ferran Alet, 2018; Li et al., 2019; Veniat et al., 2021) add new network modules or experts for new tasks.”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “an independent branch for each new task while freezing the old task parameters to preserve the old knowledge [33, 45, 43, 25, 35, 22].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Xu and Zhu [149] further proposed a Reinforced Continual Learning method, which uses a recurrent neural network trained following an actor-critic strategy to determine the optimal number of nodes and filters that should be added to each layer for each new task.”
Wanderlust: Online Continual Object Detection in the Real World (2021, methodology): “A large body of continual learning algorithms [1, 21, 29, 40, 48, 25, 46, 39, 45, 18, 24] have been developed and evaluated on image classification benchmarks such as Permuted MNIST [14], CIFAR100 [22], and ImageNet [9].”
Co-Transport for Class-Incremental Learning (2021, background): “[50, 59] retrain and expand the network for new tasks.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, methodology): “In parameter isolation methods [40, 29], model parameters are independent for each session, and one can freeze parameters for the previous session to avoid forgetting.”
Learngene: From Open-World to Your Learning Task (2021, background): “Continual learning: Continual learning (Parisi et al. 2019; De Lange et al. 2019; Kirkpatrick et al. 2017; Shin et al. 2017; Rusu et al. 2016; Rebuffi et al. 2017; Yoon et al. 2018; Xu and Zhu 2018) is also referred to as lifelong learning (Chen and Liu 2018; Aljundi, Chakravarty, and Tuytelaars 201”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “Model Zoo is a very simplistic instantiation of parameter isolation, or growing (Rusu et al., 2016; Mallya and Lazebnik, 2018; Xu and Zhu, 2018).”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, positive): “The architecture approaches [33, 44, 43, 23, 45] utilize network isolation or expansion during continual learning to improve network performance.”