Bootstrapped Meta-Learning
By Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, Satinder Singh. 2021
62 citations recorded by the source corpus.
Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent, without backpropagating through the update rule.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Matthew Riemer
- Irina Rish
- Doina Precup
- Khimya Khetarpal
- Chelsea Finn
- Joelle Pineau
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Matthew Riemer
- Trevor Darrell
- Yoshua Bengio
- Doina Precup
- Irina Rish
- Andrea Vedaldi
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2019) and meta-critics (Zhou et al., 2020; Flennerhag et al., 2021) in a true continual RL setting closer to the spirit of self-modifying policies.”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “Other approaches for meta-RL include meta-gradients (Andrychowicz et al., 2016; Finn et al., 2017; Flennerhag et al., 2022; Xu et al., 2018), which can be efficient but often suffer from instability and myopia (Flennerhag et al.”
- Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning (2022, background): “Instead of detecting changepoints, Flennerhag et al. (2022) meta-learn an exploration schedule to cope with a non-stationary reward function.”