Unsupervised Curricula for Visual Meta-Reinforcement Learning

By Allan Jabri, Kyle Hsu, Benjamin Eysenbach, Abhishek Gupta, Alexei A. Efros, Sergey Levine, Chelsea Finn. 2019

68 citations recorded by the source corpus.

In principle, meta-reinforcement learning algorithms leverage experience across many tasks to learn fast and effective reinforcement learning (RL) strategies. However, current meta-RL approaches rely on manually-defined distributions of training tasks, and hand-crafting these task distributions can be challenging and time-consuming. Can ``useful'' pre-training tasks be discovered in an unsupervised manner? We develop an unsupervised algorithm for inducing an adaptive meta-training task distribution, i.e. an automatic curriculum, by modeling unsupervised interaction in a visual environment. The task distribution is scaffolded by a parametric density model of the meta-learner's trajectory distribution. We formulate unsupervised meta-RL as information maximization between a latent task variable and the meta-learner’s data distribution, and describe a practical instantiation which alternates between integration of recent experience into the task distribution and meta-learning of the updated tasks. Repeating this procedure leads to iterative reorganization such that the curriculum adapts as the meta-learner's data distribution shifts. Moreover, we show how discriminative clustering frameworks for visual representations can support trajectory-level task acquisition and exploration in domains with pixel observations, avoiding the pitfalls of alternatives. In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that both transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient meta-learning of test task distributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “We perform the clustering for 25 epochs with a ﬁxed learning rate of 1 e − 3 , Adam optimizer (Kingma & Ba, 2014) and a batch size of 50 using the Algorithm 2 from Jabri et al. (2019).”
Meta-Learning in Neural Networks: A Survey (2020, background): “Unsupervised meta-RL variants aim to perform meta-training without manually specified rewards [232], or adapt at meta-testing to a changed environment but without new rewards [233].”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “In CARML, Jabri et al. (2019) adapt the task distribution to form a curriculum for meta-RL by maximising the information between a latent task variable and their corresponding trajectories.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “, 2019), increasing distance between the starting point and goal (Jabri et al., 2019) or using a density model to generate a task distribution for the meta learner (Florensa et al.”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, methodology): “A popular class of ACL methods learns a generative model, such as VAE [26], GAN [9, 7] and Gaussian mixture model [24, 12, 22], over the task space with density concentrated on tasks with”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “The most similar work is Jabri et al. (2019), which also considers the unsupervised application of meta-learning to RL tasks.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “Finally, leveraging information-theoretic methods to perform unsupervised task discovery in the meta-RL framework (Jabri et al., 2019) is another interesting direction for future research.”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “In this work, we focus on MDPs where only reward dynamics varies [9, 10, 12, 13, 17, 26] and only train the reward decoder (parameterized by θR) as in [56].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “Some prior works have proposed solutions to this problem by deriving goals or skills in an unsupervised manner (Gregor et al. 2017; Jabri et al. 2019).”
Replay-Guided Adversarial Environment Design (2021, methodology): “While our methods do not presuppose a goal-based setting, others have made progress here using generative modeling [12, 29], latent skill learning [14], and exploiting model disagreement [45].”