Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
By Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, Sergey Levine. 2019
1,715 citations recorded by the source corpus.
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning, Evaluation & Benchmarks, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Trevor Darrell
- Quoc V. Le
In Cognitive Neuroscience
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Yoshua Bengio
- Abhishek Gupta
- Shixiang Shane Gu
- Aviral Kumar
- Tianhe Yu
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Abhishek Gupta
- Abhinav Kumar Gupta
- Vikash Kumar
In Evaluation & Benchmarks
- Sergey Levine
- Jitendra Malik
- Abhinav Kumar Gupta
- Pieter Abbeel
- Silvio Savarese
- Trevor Darrell
- Chelsea Finn
- Dhruv Batra
In Meta & Continual Learning
- Chelsea Finn
- Eric Eaton
- Jorge Armando Mendez Mendez
- Sergey Levine
- Massimo Caccia
- Irina Rish
- Han-Jia Ye
- Razvan Pascanu
Citations in the corpus
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “Although, a recent line of work, e.g. Meta-World and RLBench (Yu et al., 2019; James et al., 2020) aim at studying multi-task learning as well as meta-learning, respective benchmarks mostly exhibit non-parametric hand-designed task variations which makes it ambiguous and not explicit how much struct”
- Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “In this work, we focus on the SAC algorithm [16], which is often considered to be the method of choice for continuous control RL [48, 47, 37].”
- CoMPS: Continual Meta Policy Search (2021, methodology): “Reaching the success threshold on the more challenging MetaWorld tasks is generally difficult, however, CoMPS still performs the best.”
- Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “While the prospect of meta-learning new solutions for continual learning is exciting, this has proven to be computationally demanding as well as requiring careful design of the task distribution [84,85].”
- Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, methodology): “To our knowledge, this is the first time lifelong transfer has been shown on the challenging Meta-World benchmark.”
- DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “On the multi-task MT10 benchmark [14] and several robotic manipulation tasks, our method learns policies with a final success rate that is 50% higher than that of SAC.”
- COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Following the same manner, we construct a third task sawyer-door-close based on the environment in Yu et al. [66], Rafailov et al. [48].”
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “A variety of new environments have been created recently to benchmark reinforcement learning algorithms [15, 34, 35, 36, 37].”
- The MAGICAL Benchmark for Robust Imitation (2020, background): “For instance, there are several challenging benchmarks for generalisation [31, 12, 13] and meta- or multi-task learning [40] in RL.”
- OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “For online multi-task transfer learning, we learn primitives with expert data from pick-and-place task and then use it to learn multi-task policy for MT10 and MT50 (from metaworld (Yu et al., 2020)), containing 10 and 50 robotic manipulation tasks which needs to be solved simultaneously.”
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, neutral): “These include Robot Learning in Homes [19], Meta-World [50], the ProcGen benchmark [10].”
- RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “We should also mention the very recently announced Meta-World project [25],”
- Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “, 2019), φ is typically related to the bonus points for destroying each type of units; in robotics manipulation (Levine et al., 2016; Li et al., 2020; Yu et al., 2019), φ is often about the distance between the robot/object and its target position; in general multi-agent games (Lowe et al.”
- Gradient Surgery for Multi-Task Learning (2020, methodology): “To answer question (2), we first consider the multi-task reinforcement learning problem and evaluate our algorithm on the recently proposed Meta-World benchmark (Yu et al., 2019).”
- Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, background): “Similarly, it can represent unobserved variables in partially observable Markov decision processes (POMDPs) [32], hidden contexts in hidden-mode Markov decision processes [12], or tasks’ variations in robotics [75, 27, 82].”
- Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “Meta-World (Yu et al., 2019) proposes several metalearning benchmarks, using up to 50 unique continuous control environments for training and testing.”
- Meta-Learning in Neural Networks: A Survey (2020, background): “medical vs satellite vs everyday images in computer vision, or putting pegs in holes vs opening doors [240] in robotics.”