Variational Auto-Regressive Gaussian Processes for Continual Learning
By Sanyam Kapoor, Theofanis Karaletsos, Thang D. Bui. 2020
33 citations recorded by the source corpus.
This paper proposes Variational Auto-Regressive Gaussian Process (VAR-GP), a principled Bayesian updating mechanism to incorporate new data for sequential tasks in the context of continual learning. It relies on a novel auto-regressive characterization of the variational distribution and inference is made scalable using sparse inducing point approximations. Experiments on standard continual learning benchmarks demonstrate the ability of VAR-GPs to perform well at new tasks without compromising performance on old ones, yielding competitive results to state-of-the-art methods. In addition, we qualitatively show how VAR-GP improves the predictive entropy estimates as we train on new tasks. Further, we conduct a thorough ablation study to verify the effectiveness of inferential choices.
Subfields: Probabilistic & Bayesian Methods, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Probabilistic & Bayesian Methods
- Jun Zhu
- Yarin Gal
- Yee Whye Teh
- Eric Eaton
- Tim G. J. Rudner
- Jorge Armando Mendez Mendez
- Qixuan Feng
- Liyuan Wang
In Meta & Continual Learning
- Jorge Armando Mendez Mendez
- Eric Eaton
- Xingxing Zhang
- Liyuan Wang
- Hang Su
- Jun Zhu
- Qixuan Feng
- Freddie Bickford-Smith
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The second is online variational inference (VI) [55], [56], [57], [58], [59], [60].”
- Continual Learning via Sequential Function-Space Variational Inference (2023, background): “The success of variational autoregressive GP s ( VAR - GP ; Kapoor et al., 2021) on continual learning for task sequences with image inputs gives reason for inclusion where relevant in Section 5.”
- Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[48] performed regularization from another perspective.”
- How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “A distinct approach by Kapoor et al. (2021) trains a variational GP using sparse sets of inducing points per task.”