RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

By Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel. 2016

1,188 citations recorded by the source corpus.

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “To evaluate MAML on reinforcement learning problems, we constructed several sets of tasks based off of the simulated continuous control environments in the rllab benchmark suite (Duan et al., 2016a).”
Learning with Latent Language (2017, methodology): “…but once a description is chosen the score 4In RL, the contribution of L3 is orthogonal to that of meta-learning—one could use a technique like RL2 (Duan et al., 2016) to generate candidate descriptions more efficiently, or MAML (Finn et al., 2017) rather than zero-shot reward as the training…”
One-Shot Imitation Learning (2017, background): “…been studied for image recognition (Vinyals et al., 2016; Koch, 2015; Santoro et al., 2016; Ravi & Larochelle, 2017), generative modeling (Edwards & Storkey, 2017; Rezende et al., 2016), learning “fast” reinforcement learning agents with recurrent policies (Duan et al., 2016; Wang et al., 2016).”
Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “[4] and Duan et al.[5] demonstrate that optimization algorithms can be learned by RNNs, making it possible that a Universal Planning Networks-like agent could be entirely learned—assuming a very expressive model space—including the internal optimization steps.”
Shaking the foundations: delusions in sequence models for interaction and control (2021, methodology): “Meta-learning and “counterfactual teaching” We can learn a sequential model for control using memory-based meta-learning [9, 24] (as was done to train e.”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “The concept of having one neural network interact with a second neural network to aid the learning process, or learning to learn or meta-learning [21, 41] has attracted much attention in recent years [1, 13, 14, 18, 32, 38, 52].”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, methodology): “In fact, much of the meta learning literature focuses on the general reinforcement learning tasks [7, 28].”
Meta-Gradient Reinforcement Learning (2018, background): “Duan et al. [2016] and Wang et al. [2016a] proposed to learn a recurrent meta-policy that itself learns to solve the reinforcement learning problem, so that the recurrent policy can generalise into new tasks faster than learning the policy from scratch.”
Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, background): “Meta-learners developed so far include recurrent models [6, 10, 19, 26], metrics [12, 25], or optimizers [2, 7, 16, 18].”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “Related ideas were also discussed in [27, 11, 117, 20].”
Causal Reasoning from Meta-reinforcement Learning (2019, methodology): “We specifically adopt the “meta-reinforcement learning” method introduced previously (Duan et al., 2016; Wang et al., 2016), in which a recurrent neural network (RNN)-based agent is trained with model-free reinforcement learning (RL).”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, background): “In the ﬁrst approach (Santoro et al., 2016; Duan et al., 2016; Wang et al., 2016; Mishra et al., 2017), there is a meta-learner model g with parameters φ which takes as input the dataset D T for a particular task T and a new test input x (cid:63) , and outputs the estimated output ˆ y (cid:63) for…”
Meta-learning autoencoders for few-shot prediction (2018, methodology): “[18] utilizes a neural Turing machines for few-shot learning, and [4, 24] learn fast reinforcement learning agents with recurrent policies using memory-augmented nets.”