Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint.

By Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, Alex Mott. 2020

44 citations recorded by the source corpus.

Subfields: Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “Online structured Laplace (Ritter et al., 2018) and second-order loss approximation (Yin et al., 2020a) respectively use Kronecker-factored and low-rank Hessians.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, background): “Comparable assumptions are also made in existing studies on continual learning [55].”
Continual Learning through Retrieval and Imagination (2022, methodology): “In this section, we analyze the generalization of our learning framework based on robustness theory in machine learning (Bellet et al. 2015; Mohri et al. 2009; Yin et al. 2020; Guo et al. 2022).”
Architecture Matters in Continual Learning (2022, background): “(Zenke et al., 2017; Kirkpatrick et al., 2017; Mirzadeh et al., 2021b; Yin et al., 2020) identify essential knowledge from the past, often in the form of parameters or features, and modify the training objective of the new task to preserve the learned knowledge.”
Wide Neural Networks Forget Less Catastrophically (2021, methodology): “Regularization-based methods (Rebuffi et al., 2016; Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018; Aljundi et al., 2018; Yin et al., 2020) attempt to identify important parameters or features of the past tasks, and modify the training objective of the new task such that those par”
Provable Continual Learning via Sketched Jacobian Approximations (2021, background): “Some recent theoretical developments include (Yin et al., 2020; Bennani et al., 2020; Alquier et al., 2017; Doan et al., 2021).”
Formalizing the Generalization-Forgetting Trade-off in Continual Learning (2021, methodology): “For example, common solutions to the catastrophic forgetting issue include (1) representation-driven approaches [49, 25], (2) regularization approaches [27, 2, 34, 16, 48, 48, 24, 37, 10, 43], and (3) memory/experience replay [31, 32, 11, 12, 17].”
Provable Lifelong Learning of Representations (2021, methodology): “[Yin et al., 2020] studies the optimization and generalization properties of the regularization-based method by analyzing the loss landscape.”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, methodology): “Among pioneer methods to alleviate the catastrophic forgetting, we can name regularization-based methods that limit the drift in important parameters of features of past tasks (Kirk-patrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2017; Yin et al., 2020).”
Provable and Efficient Continual Representation Learning (2022, background): “We also emphasize that there are several interesting works on the theory of continual learning such as Lee et al. (2021); Doan et al. (2021); Bennani et al. (2020); Mirzadeh et al. (2020); Yin et al. (2020).”