Unsupervised Meta-Learning for Reinforcement Learning

By Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine. 2018

114 citations recorded by the source corpus.

Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “A common theme in research is to try to exploit a shared structure in a distribution of environments, to quickly identify and adapt to previously unseen cases [55, 94, 47, 54] There are works that directly treat meta-learning as identification of a dynamics model of the environment [19] while others”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Much recent met-alearning work [6, 26, 27, 31, 37, 38, 53, 56, 60, 64, 79] focuses on domains where achieving zero-shot generalization is unclear and follow a common paradigm which assumes that the shared representation of monolithic architectures (with the exception of concurrent work [2]) can be…”
POET: open-ended coevolution of environments and their optimized solutions (2019, background): “[19] note that the performance of meta-learning algorithms”
Concurrent Meta Reinforcement Learning (2019, unknown): “…have demonstrated impressive performance in the areas of reinforcement learning Finn et al. (2017), learning to explore Gupta et al. (2018b), unsupervised reinforcement learning Gupta et al. (2018a) and multiagent and non-stationary environments Al-Shedivat et al. (2017); Raileanu et al. (2018).”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “…learning performance (Mishra et al., 2018; Duan et al., 2017; Snell et al., 2017), learning to explore (Stadie et al., 2018), unsupervised learning (Gupta et al., 2018; Hsu et al., 2018), few-shot model adaptation (Nagabandi et al., 2018), and improving the optimizers (Andrychowicz et al., 2016;…”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “There have been many other important recent papers investigating how much agents can learn when motivated by curiosity alone [19, 128, 126, 147, 42, 56, 127, 150].”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, methodology): “We consider variants of our method to ablate the role of design decisions related to task acquisition and joint training: 4) pipelined (most similar to [25]) – task acquisition with a contextual policy, followed by meta-RL with RL(2); 5) online discriminator – task acquisition with a purely discrimi”
Meta Reinforcement Learning with Task Embedding and Shared Policy (2019, methodology): “However, we observe that simply adapting a single variable is not enough to represent a task in our experiments (conducted in a more challenging way than [Gupta et al. , 2018b]).”
ProMP: Proximal Meta-Policy Search (2018, background): “…approach to be promising, demonstrating that Meta-RL allows agents to obtain a diverse set of skills, attain better exploration strategies, and learn faster through meta-learned dynamics models or synthetic returns (Duan et al., 2016; Xu et al., 2018; Gupta et al., 2018b; Saemundsson et al., 2018).”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “In practice, the performance of meta-learning algorithms depends on the user-specified meta-training task distribution [Gupta et al., 2018].”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Some methods use distributions of tasks (Hausman et al., 2018; Gupta et al., 2018b) to obtain diverse skills, while other methods propose to maximize the mutual information between the latent variable and states (Eysenbach et al., 2019; Gupta et al., 2018a; Florensa et al., 2017) or final states…”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “Some meta-learning methods adapt using both labeled and unlabeled data, such as [53, 71, 46, 2, 39, 4], or consider the setting where task groupings are not known, such as [23, 30, 24, 59], though these works do not focus on the same setting of distribution shift.”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “In this work, we focus on MDPs where only reward dynamics varies [9, 10, 12, 13, 17, 26] and only train the reward decoder (parameterized by θR) as in [56].”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, methodology): “[15] propose an unsupervised meta-RL technique that computes minimax regret against a diverse task distribution learned with the DIAYN algorithm [10].”