Discovering Reinforcement Learning Algorithms

By Junhyuk Oh, Matteo Hessel, Wojciech M. Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, David Silver. 2020

149 citations recorded by the source corpus.

Reinforcement learning (RL) algorithms update an agent's parameters according to one of several possible rules, discovered manually through years of research. Automating the discovery of update rules from data could lead to more efficient algorithms, or algorithms that are better adapted to specific environments. Although there have been prior attempts at addressing this significant scientific challenge, it remains an open question whether it is feasible to discover alternatives to fundamental concepts of RL such as value functions and temporal-difference learning. This paper introduces a new meta-learning approach that discovers an entire update rule which includes both 'what to predict' (e.g. value functions) and 'how to learn from it' (e.g. bootstrapping) by interacting with a set of environments. The output of this method is an RL algorithm that we call Learned Policy Gradient (LPG). Empirical results show that our method discovers its own alternative to the concept of value functions. Furthermore it discovers a bootstrapping mechanism to maintain and use its predictions. Surprisingly, when trained solely on toy environments, LPG generalises effectively to complex Atari games and achieves non-trivial performance. This shows the potential to discover general RL algorithms from data.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2017), discovering RL algorithms (Kirsch et al., 2019; Oh et al., 2020), learning what objective to learn (Xu et al.”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, methodology): “Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem of automatically discovering general-purpose RL algorithms (Kirsch, van Steenkiste, and Schmidhuber 2020; Alet et al. 2020; Oh et al. 2020).”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, background): “…the goal of achieving fast adaptation to changes in distribution or a new task in a reinforcement-learning agent (Duan et al., 2016; Mishra et al., 2017; Wang et al., 2016; Nichol et al., 2018; Xu et al., 2018; Houthooft et al., 2018; Kirsch et al., 2019; Botvinick et al., 2019; Oh et al., 2020).”
Transformers are Meta-Reinforcement Learners (2022, background): “Another branch of methods learns the entire system using memory-based architectures (Ortega et al., 2019; Wang et al., 2017; Duan et al., 2016; Ritter et al., 2018a) or generating update rules by discovery (Oh et al., 2020) or evolution (Co-Reyes et al., 2021).”
Evolving Reinforcement Learning Algorithms (2021, background): “Learning RL algorithms or their components, such as a reward bonus or value update function, has been studied previously with meta-gradients (Kirsch et al., 2020; Chebotar et al., 2019; Oh et al., 2020), evolutionary strategies (Houthooft et al., 2018), and RNNs (Duan et al., 2016).”
Bootstrapped Meta-Learning (2021, background): “More generally, the update rule can be an arbitrary parameterised function (Hochreiter et al., 2001; Andrychowicz et al., 2016; Kirsch et al., 2019; Oh et al., 2020), or the function itself can be meta-learned jointly with its parameters (Alet et al., 2020; Real et al., 2020).”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “The metagradient approach has also been applied, ofﬂine and over multiple lifetimes, to black-box parameterisations, via deep neural networks, of the entire RL algorithm [2, 19] and [24] (contemporaneous work); evolutionary approaches have also been applied [16].”