Hindsight policy gradients

By Paulo E. Rauber, Filipe Wall Mutz, J. Urgen Schmidhuber. 2017

76 citations recorded by the source corpus.

Goal-conditional policies allow reinforcement learning agents to pursue specific goals during different episodes. In addition to their potential to generalize desired behavior to unseen goals, such policies may also help in defining options for arbitrary subgoals, enabling higher-level planning. While trying to achieve a specific goal, an agent may also be able to exploit information about the degree to which it has achieved alternative goals. Reinforcement learning agents have only recently been endowed with such capacity for hindsight, which is highly valuable in environments with sparse rewards. In this paper, we show how hindsight can be introduced to likelihood-ratio policy gradient methods, generalizing this capacity to an entire class of highly successful algorithms. Our preliminary experiments suggest that hindsight may increase the sample efficiency of policy gradient methods.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling (Andrychowicz et al., 2017; Rauber et al., 2019; Ghosh et al., 2021; Paster et al., 2021) and reward conditioning (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019b) to reinterpret all…”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Another important area of research builds off goal conditioned value functions and policies while leveraging the concept of hindsight to learn off-policy about goals that were achieved, even if they did not match the original goal of the policy (Andrychowicz et al., 2017; Rauber et al., 2017; Li et ”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “Some preliminary results with vanilla Policy Gradients were presented by Rauber et al. (2017), but this approach needs to be tested on more challenging environments like the ones proposed in this report.”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Goal relabeling has been explored in prior work [18, 2, 37, 25, 35].”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “…is simple: sample a segment bounded by time indices t 1 and t 2 from any trajectory, then construct training data by taking its first state ( s t 1 ) and action ( a t 1 ) , and compute the values of d r and d h for it in hindsight [Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2017].”
Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following (2019, background): “, 2019), or they adapt goal relabelling to policy gradient methods (Rauber et al., 2019).”
Learning One Representation to Optimize All Rewards (2021, background): “Pacman, where an episode ends when the agent is captured by a monster [RUMS18].”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling [2, 48, 18] and reward-conditioning [52, 55, 32] to reinterpret all past experience as useful demonstrations with proper contextualization.”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “These extensions deal with automatic curriculum learning [28], dynamic goals [15], or they adapt goal rela-belling to policy gradient methods [34].”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “A similar observation of hindsight relabelling was originally made by Kaelbling [1993], more recently popularized in the deep RL literature [Andrychowicz et al., 2017], for learning with off-policy value-based methods and policy-gradient methods [Rauber et al., 2017].”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, unknown): “This is similar to hindsight policy gradient [28] and experience replay [2] and other counterfactual RL works [6, 9, 21].”
Hindsight Credit Assignment (2019, background): “[15] extend the same intuition to policy gradient algorithms, with goal-conditioned policies.”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, background): “For all past pairs of times (time1 time2) it can retrospectively [1, 35] invent additional, consistent, vector-valued command inputs for itself, indicating tasks such as: achieve the already observed rewards/costs between time1 and time2 .”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, background): “Both Andrychowicz et al. (2017) and Rauber et al. (2017) leverage the idea of using hindsight information to learn goal-conditioned policies.”
Learning Interactive Real-World Simulators (2023, methodology): “To use the simulator for long-horizon tasks, we draw inspiration from hindsight relabeling [34].”