Learning to reinforcement learn

By Jane X. Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, R´emi Munos, Charles Blundell, Dharshan Kumaran, Matthew Botvinick. 2016

1,097 citations recorded by the source corpus.

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “, 2016) specifically, and recurrent meta-learning models in general, represent a more broadly applicable class of methods that can be used for other tasks such as reinforcement learning (Duan et al., 2016b; Wang et al., 2016).”
One-Shot Imitation Learning (2017, background): “…been studied for image recognition (Vinyals et al., 2016; Koch, 2015; Santoro et al., 2016; Ravi & Larochelle, 2017), generative modeling (Edwards & Storkey, 2017; Rezende et al., 2016), learning “fast” reinforcement learning agents with recurrent policies (Duan et al., 2016; Wang et al., 2016).”
Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “’s “Learning to Reinforcement Learn,” the authors claim to have produced a neural network that implements its own optimization procedure.[28] Specifically, the authors argue that the ability of their network to solve extremely varied environments without explicit retraining for each one means that t”
Shaking the foundations: delusions in sequence models for interaction and control (2021, methodology): “We can learn a sequential model for control using memory-based meta-learning [9, 24] (as was done to train e.g. GPT-3 [5]).”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “The concept of having one neural network interact with a second neural network to aid the learning process, or learning to learn or meta-learning [23, 49] has attracted much attention in recent years [1, 62, 14, 19, 35, 45, 15].”
How to train your MAML (2018, background): “Recent work in meta-learning has produced state of the art results in a variety of settings (Wang et al., 2016; Ba et al., 2016; Zoph & Le, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016; Li & Malik, 2016; Li et al., 2017; Antoniou et al 2017; Brock et al., 2017; Munkhdalai & Yu, 2017).”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “Meta-learning: Our study is also related to meta-learning [5, 1, 20, 34].”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “In fact, much of the meta learning literature focuses on the general reinforcement learning tasks [7, 26].”
Meta-Gradient Reinforcement Learning (2018, background): “[Wang et al., 2016b].”
Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, background): “Meta-learners developed so far include recurrent models [6, 10, 19, 26], metrics [12, 25], or optimizers [2, 7, 16, 18].”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For instance, context-based meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned [8, 9, 10, 11].”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “The approach we’ve taken is directly based on RL(2) [27, 116] where a general-purpose optimization algorithm trains a model augmented with memory to perform independent learning algorithm in the inner loop.”
Causal Reasoning from Meta-reinforcement Learning (2019, methodology): “Following the approach of (Duan et al., 2016; Wang et al., 2016), we parameterize the entire learning algorithm as a recurrent neural network (RNN), and we train the weights of the RNN with model-free reinforcement learning (RL).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Authors of [132] addressed this issue through training a recurrent neural network on a training set of interrelated tasks, where the network input includes the action selected in addition to the reward received in the previous time step.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “5) A2C (Wang et al., 2016a; Wu et al., 2017): : Some recent works found that the asynchrony in A3C does not necessarily lead to improved performance compared to the synchronous version: A2C.”
Conditional Neural Processes (2018, methodology): “Recently meta-learning has been applied to a wide range of tasks like RL (Wang et al., 2016; Finn et al., 2017) or program induction (Devlin et al., 2017).”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, background): “In the ﬁrst approach (Santoro et al., 2016; Duan et al., 2016; Wang et al., 2016; Mishra et al., 2017), there is a meta-learner model g with parameters φ which takes as input the dataset D T for a particular task T and a new test input x (cid:63) , and outputs the estimated output ˆ y (cid:63) for…”