Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

By Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Joshua Tobin, Maciek Chociej, Peter Welinder and 2 more. 2018

642 citations recorded by the source corpus.

The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “In single-agent RL, standard environments such as the Arcade Learning Environment (Bellemare et al., 2013) and MuJoCo (Plappert et al., 2018) have facilitated rapid progress.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “To this end, we include experiments on continuous control tasks in the OpenAI Gym benchmark (Plappert et al., 2018) (HalfCheetah-v2, Hopper-v2, Ant-v2, Walker2d-v2).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “For more details, we refer the readers to [Plappert et al., 2018].”
The StarCraft Multi-Agent Challenge (2019, methodology): “can be overly simple or tuned to the proposed algorithms. In single-agent RL, standard environments such as the Arcade Learning Environment (Bellemare et al., 2013), or MuJoCo for continuous control (Plappert et al., 2018), have enabled great progress. In this paper, we aim to follow this successful”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “Sunehag et al., 2017), we introduce the StarCraft Multi-Agent Challenge (SMAC)1. In single-agent RL, standard environments such as the Arcade Learning Environment (Bellemare et al., 2013) and MuJoCo (Plappert et al., 2018) have facilitated rapid progress. While some multi-agent testbeds have emerged”
Learning dexterous in-hand manipulation (2018, methodology): “There has also been signiﬁcant progress in learning complex in-hand dexterous manipulation [49, 6] and even tool use [50] using deep reinforcement learning but those approaches were only evaluated in simulation as well.”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “To match the original work as closely as possible, this comparison was based off of the OpenAI baselines code [34] and we compare on the same Fetch robotics tasks.”
Data-Efficient Hierarchical Reinforcement Learning (2018, methodology): “They are a natural choice for a generally applicable HRL method and have therefore appeared as components of other HRL methods [43, 22, 30, 24].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…stable improvement and solving tasks without large degradation or sub-optimal convergence. we limit the batch size at each deployment as only 25k. Figure 3 presents the reaching tasks with Fetch robot and 20-DoF shadow hand (Plappert et al., 2018), and the experimental results in both environments.”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, methodology): “, 2016): Half-Cheetah, Fetch-Push (Plappert et al., 2018) and Reacher.”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “They are also very sensitive to the frequency of taking actions, limiting their applicability to robot control [Plappert et al., 2018].”
Improving Generalization in Meta Reinforcement Learning using Learned Objectives (2019, background): “However, while prior work found that learning rules could be meta-learned that generalize to slightly different environments or goals (Finn et al., 2017; Plappert et al., 2018; Houthooft et al., 2018), generalization to entirely different environments remains an open problem.”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, positive): “…RL agent. c) Hindsight relabeling for goal-conditioned reinforcement learning: The relabeling approach for goal-conditioned reinforcement learning [38] is used in tabular [26] and continuous [4] settings, where the desired goals are relabeled with achieved goals to generate positive examples in…”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, neutral): “• Visual Fetch Reach (OpenAI Gym [39]).”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, background): “We focus on multi-goal environments and goal-conditioned policies (Foster and Dayan, 2002; Schaul et al., 2015; Plappert et al., 2018).”
Residual Policy Learning (2018, methodology): “We conﬁrm the results reported in previous work [31] that learning from scratch with DDPG and HER works well in this domain, converging to a success rate of nearly 1.0 after roughly 2 million simulator steps.”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, methodology): “a two-fingered parallel jaw gripper based on OpenAI’s FetchPickAndPlace [59] and MuJoCo physics engine [60].”