Learning where to learn: Gradient sparsity in meta and continual learning

By Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, João Sacramento. 2021

60 citations recorded by the source corpus.

Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Continual Learning via Local Module Composition (2021, background): “Continual-meta learning is often confused with its counterpart meta-continual learning [40, 11, 93], in which algorithm are learning to continually learn.”
New Insights on Reducing Abrupt Representation Change in Online Continual Learning (2021, background): “The single-head assumption is further studied in task-agnostic continual learning settings (He et al., 2019; Caccia et al., 2020; Ostapenko et al., 2021; Von Oswald et al., 2021; Lesort et al., 2019) in which the task-boundary assumption, amongst others, is also relaxed.”
Transformers learn in-context by gradient descent (2022, background): “…the solution found can be approximated well when only adapting the output layer i.e. learning a linear model on a meta-learned deep data representations (Finn et al., 2017; Finn & Levine, 2018; Gordon et al., 2019; Lee et al., 2019; Rusu et al., 2019; Raghu et al., 2020; von Oswald et al., 2021).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “An extension to the technique of Gupta et al. (2020a, from the task-aware setting) for lifelong learning via meta-learning trains an additional binary mask that determines which parameters to learn for each task, leading to sparse gradients (von Oswald et al., 2021).”