Meta-Gradient Reinforcement Learning with an Objective Discovered Online
By Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, David Silver. 2020
90 citations recorded by the source corpus.
Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Hado van Hasselt
- Pieter Abbeel
- David Silver
- Matteo Hessel
- Doina Precup
- Satinder Singh
- Chelsea Finn
In Statistical and Nonlinear Physics
- Doina Precup
- Wojciech M. Czarnecki
- David Silver
- Satinder Singh
- Irina Rish
- Yutian Chen
- Junhyuk Oh
- Matteo Hessel
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…(Kirsch, van Steenkiste, and Schmidhuber, 2019; Oh, Hessel, Czarnecki, Xu, van Hasselt, Singh, and Silver, 2020), learning what objective to learn (Xu et al., 2020), and other approaches moving more and more towards open-ended learning (Wang, Lehman, Rawal, Zhi, Li, Clune, and Stanley, 2020b).”
- Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “2020), meta-learning hyperparameters (Sutton 1992; Xu, van Hasselt, and Silver 2018) or meta-learning objective functions corresponding to the learning algorithm (Houthooft et al. 2018; Kirsch, van Steenkiste, and Schmidhuber 2020; Xu et al. 2020; Oh et al. 2020; Bechtle et al. 2021).”
- Discovering Reinforcement Learning Algorithms (2020, methodology): “This approach has been applied to meta-learn various forms of algorithmic components such as the discount factor [38], intrinsic rewards [41], auxiliary tasks [33], returns [36], auxiliary policy updates [42], off-policy corrections [39], and update target [37].”