Improving Generalization in Meta Reinforcement Learning using Learned Objectives

By Louis Kirsch, Sjoerd van Steenkiste, J. Urgen Schmidhuber. 2019

132 citations recorded by the source corpus.

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “, 2019) or the objective function in reinforcement learning (Houthooft et al., 2018; Kirsch et al., 2019; Alet et al., 2019); in these works, all other algorithmic aspects remain hand-designed.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2017), discovering RL algorithms (Kirsch et al., 2019; Oh et al., 2020), learning what objective to learn (Xu et al.”
Bootstrapped Meta-Learning (2021, background): “More generally, the update rule can be an arbitrary parameterised function (Hochreiter et al., 2001; Andrychowicz et al., 2016; Kirsch et al., 2019; Oh et al., 2020), or the function itself can be meta-learned jointly with its parameters (Alet et al., 2020; Real et al., 2020).”
Model-based Adversarial Meta-Reinforcement Learning (2020, background): “MetaGenRL [19] proposes to meta-learn objective functions to generalize to different environments.”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, methodology): “MetaGenRL [21] meta-learns an RL objective to train a randomly initialized policy on a test task.”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, background): “…into the meta-level optimization (Finn et al., 2017a; Nagabandi et al., 2019; Rothfuss et al., 2018; Zintgraf et al., 2019; Gupta et al., 2018; Mendonca et al., 2019; Yang et al., 2019), potentially using a learned loss function (Houthooft et al., 2018; Bechtle et al., 2019; Kirsch et al., 2020).”
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, background): “Recently, meta-learning [14] has become topical as a paradigm to accelerate RL by learning aspects of the learning strategy, for example, learning fast adaptation strategies [7, 30, 31], losses [3, 15, 17, 36], optimisation strategies [6], exploration strategies [11], hyperparameters [40, 42], and i”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem of automatically discovering general-purpose RL algorithms (Kirsch, van Steenkiste, and Schmidhuber 2020; Alet et al. 2020; Oh et al. 2020).”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, background): “…the goal of achieving fast adaptation to changes in distribution or a new task in a reinforcement-learning agent (Duan et al., 2016; Mishra et al., 2017; Wang et al., 2016; Nichol et al., 2018; Xu et al., 2018; Houthooft et al., 2018; Kirsch et al., 2019; Botvinick et al., 2019; Oh et al., 2020).”
Evolving Reinforcement Learning Algorithms (2021, background): “, 2018), or learns a domain invariant policy update rule that can generalize to new environments (Kirsch et al., 2020).”
Discovering Reinforcement Learning Algorithms (2020, methodology): “Most recently, MetaGenRL [17] was proposed to meta-learn a domain-invariant policy update rule, capable of generalizing from a few MuJoCo environments to other MuJoCo environments.”
Meta-learning curiosity algorithms (2020, background): “Concurrent to our work, Kirsch et al. (2019) also show generalization capabilities between environments similar to ours (lunar lander, hopper and half-cheetah).”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “…targets v.s. Discovering loss : Our ﬁrst experiment compares the performance of online objective discovery between a meta-network that represents the target and a meta-network that represents the loss , similarly to prior work in ofﬂine, multi-lifetime setups such as ML 3 [2], MetaGenRL [19].”