Hindsight Experience Replay
By Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, Pieter Abbeel, Wojciech Zaremba. 2017
2,808 citations recorded by the source corpus.
Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.
Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Vikash Kumar
- Shixiang Shane Gu
- Dieter Fox
- Jan Peters
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Abhishek Gupta
- Shimon Whiteson
- Yuxi Li
Citations in the corpus
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[30, 2]).”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “Nair et al. (2018) combine these methods with hindsight experience replay (Andrychowicz et al., 2017).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “…their probabilities to zero when sampling actions from πθ during training; this periodic updating policy πψ is inspired by off-policy Q-learning algorithms (Andrychowicz et al., 2017), providing the policy πθ with an additional constraint that balances between the benefits of containing more…”
- Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “A promising way to scale up supervision is Hindsight Experience Replay (HER) [3], which allows agents to learn from orders of magnitude more data without supervision.”
- Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling (Andrychowicz et al., 2017; Rauber et al., 2019; Ghosh et al., 2021; Paster et al., 2021) and reward conditioning (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019b) to reinterpret all…”
- DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Other approaches such as SAC-X [Riedmiller et al., 2018] or HER [Andrychowicz et al., 2017] take advantage of Multi-task RL by learning auxiliary tasks in order to help learning a main task.”
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, methodology): “Recently (Andrychowicz et al., 2017) proposed the Hindsight Experience Replay (HER) algorithm for learning a family of goal directed policies.”
- Reinforcement Learning in Healthcare: A Survey (2019, background): “A number of effective approaches have been proposed, such as the hindsight experience replay [313], the unsupervised auxiliary learning [314], the imagination-augmented learning [315], and the reward shaping [234].”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Prioritized Experience Replay (Schaul et al., 2015b): Prioritizes memory replay according to td-error ; can be added to off-policy methods.”
- The Challenges of Continuous Self-Supervised Learning (2022, methodology): “The training data stream is created by sampling images at random from the four datasets {Dp1 , Dp2 , Dp3 , Dp4 } where [ p1 , p2 , p3 , p4] is a permutation of the sequence [1 , 2 , 3 , 4] .”
- Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “While sampling strategies have not demonstrated significant success for supervised learning problems, the reinforcement learning community has seen more benefit from these approaches, for example, prioritized experience replay (Schaul et al., 2016) and hindsight replay (Andrychowicz et al., 2017).”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Another important area of research builds off goal conditioned value functions and policies while leveraging the concept of hindsight to learn off-policy about goals that were achieved, even if they did not match the original goal of the policy (Andrychowicz et al., 2017; Rauber et al., 2017; Li et ”
- Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, methodology): “Methods using exemplars have shown great success in different continual learning (Rebuffi et al., 2017; Castro et al., 2018; Chaudhry et al., 2019) and reinforcement learning (Schaul et al., 2016; Andrychowicz et al., 2017) tasks.”
- Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, neutral): “4c shows GC-DQL GC-DQL-sep CAPs (ours) (a) Birds-eye view for each approach of the resulting trajectories when attempting to go to either the red, green, or blue goal location (five attempts per goal location).”
- Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Our work builds on goal-conditioned RL [17, 18, 19, 20], which has been explored in the context of reward-free learning [21], learning with sparse rewards [18], large scale generalizable imitation learning [22], and hierarchical RL [15].”
- Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “The previous state-of-the-art algorithm from Nair et al. [2018] combines HER [Andrychowicz et al., 2017] with BC and a few techniques: 1) an additional replay buffer filled with demonstrations, 2) an additional behavioral cloning loss for the policy, 3) a Q-filter for non-optimal demonstrations, 4)…”
- Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “In concurrent work [32], this approach was combined with hindsight experience replay [3].”
- Reinforcement Learning with Augmented Data (2020, methodology): “Hindsight Experience Replay [31] applies the idea of re-labeling trajectories with terminal states as fictitious goals, improving the ability of goal-conditioned RL to learn quickly with sparse rewards.”
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “HER (Andrychowicz et al., 2017) exploits information about the observation space by goal and reward relabeling, which can be viewed as a way to perform data augmentation.”
- IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “cent work has extended reinforcement learning [4, 27, 31] and imitation learning [7, 24] to condition on goal observations, enabling improved sample efficiency.”