Hindsight Experience Replay

By Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, Pieter Abbeel, Wojciech Zaremba. 2017

2,808 citations recorded by the source corpus.

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Pieter Abbeel
  3. Yoshua Bengio
  4. Chelsea Finn
  5. Ruslan Salakhutdinov
  6. Mohit Bansal
  7. Quoc V. Le
  8. Trevor Darrell

In Control and Systems Engineering

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Abhishek Gupta
  5. Vikash Kumar
  6. Shixiang Shane Gu
  7. Dieter Fox
  8. Jan Peters

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shixiang Shane Gu
  6. Abhishek Gupta
  7. Shimon Whiteson
  8. Yuxi Li

Citations in the corpus