Bayesian Incremental Learning for Deep Neural Networks
By Max Kochurov, Timur Garipov, Dmitrii Podoprikhin, Dmitry Molchanov, Arsenii Ashukha, Dmitry P. Vetrov. 2018
24 citations recorded by the source corpus.
In industrial machine learning pipelines, data often arrive in parts. Particularly in the case of deep neural networks, it may be too expensive to train the model from scratch each time, so one would rather use a previously learned model and the new data to improve performance. However, deep neural networks are prone to getting stuck in a suboptimal solution when trained on only new data as compared to the full dataset. Our work focuses on a continuous learning setup where the task is always the same and new parts of data arrive sequentially. We apply a Bayesian approach to update the posterior approximation with each new piece of data and find this method to outperform the traditional approach in our experiments.
Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alessandro Achille
- Stefano Soatto
- Tal Hassner
- Cuong V Nguyen
- Badri N. Patro
- Vinay P. Namboodiri
- Vinod Kumar Kurmi
- Vijay Mahadevan
In Probabilistic & Bayesian Methods
- Alessandro Achille
- Patrick McClure
- Charles Y. Zheng
- John Rogers-Lee
- Satrajit S. Ghosh
- Vinay P. Namboodiri
- Francisco Pereira
- Cuong V Nguyen
Citations in the corpus
- Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “More expressive classes of variational distributions were also considered, including channel factorized Gaussian [22], multiplicative normalizing flow [22], or structured Laplace approximations [33].”
- Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, background): “In [17], a Bayesian framework is discussed for the incremental learning problem.”
- Meta-Learning with Shared Amortized Variational Inference (2020, unknown): “The Bayesian approach is particularly interesting, since it provides a coherent framework to reason about model uncertainty, not only in small sample-size settings, but also others such as incremental learning (Kochurov et al., 2018), and ensemble learning (Gal & Ghahramani, 2016).”
- Distributed Weight Consolidation: A Brain Segmentation Case Study (2018, methodology): “Recently, these methods have been reinterpreted from a Bayesian perspective [19, 15].”