Natural Variational Continual Learning

By Hanna Tseran. 2018

Subfields: Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

1. Stefano Soatto
2. Alessandro Achille
3. Mohamed Elhoseiny
4. Marcus Rohrbach
5. Sayna Ebrahimi
6. Tal Hassner
7. Gido M. van de Ven
8. Cuong V Nguyen

In Probabilistic & Bayesian Methods

1. Sayna Ebrahimi
2. Trevor Darrell
3. Alessandro Achille
4. Gido M. van de Ven
5. Dana Rezazadegan
6. Vladimir Makarenkov
7. Huaping Liu
8. Abbas Khosravi

Citations in the corpus

• Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “Improved training procedures have also been developed for this type of approximate Bayesian continual learning through the use of natural gradients [10, 47], fixed-point updates [52], and local approximation [6].”
• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “This is attributed to the fundamental difference between the two approximation strategies [55], [61].”
• Continual Learning with Recursive Gradient Optimization (2022, methodology): “…goal of these methods are variants of Lk(θ) +λFRLLk (θ), derived from Bayesian posterior approximation with Gaussian prior (Kirkpatrick et al., 2017; Nguyen et al., 2018) or approximation of the KL-divergence used in natural gradient descent (Amari, 1998; Ritter et al., 2018; Tseran et al., 2018).”
• Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “Recent work has also combined Bayesian continual learning with natural gradient descent (Osawa et al., 2019; Tseran et al., 2018), and in this case a relatively high value of λ = 100 was similarly found to maximize performance (Osawa et al., 2019).”
• Optimal Continual Learning has Perfect Memory and is NP-hard (2020, background): “, 2019) as well as Bayesian and variationally Bayesian approaches (e.g. Nguyen et al., 2017; Tseran et al., 2018; Moreno-Munoz et al., 2019; Titsias et al., 2019).”
• Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “For Bayesian methods, we compare against VCL and its variations named as VCL with Variational Adam (VCL-Vadam), VCL with Adam and Gaussian natural gradients (VCL-GNG).”
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[505], [506] proposed variational CL (VCL) to deal with uncertainty and showed the effectiveness of such a approach.”