Generalized Hindsight for Reinforcement Learning

By Alexander C. Li, Lerrel Pinto, Pieter Abbeel. 2020

78 citations recorded by the source corpus.

One of the key reasons for the high sample complexity in reinforcement learning (RL) is the inability to transfer knowledge from one task to another. In standard multi-task RL settings, low-reward data collected while trying to solve one task provides little to no signal for solving that particular task and is hence effectively wasted. However, we argue that this data, which is uninformative for one task, is likely a rich source of information for other tasks. To leverage this insight and efficiently reuse data, we present Generalized Hindsight: an approximate inverse reinforcement learning technique for relabeling behaviors with the right tasks. Intuitively, given a behavior generated under one task, Generalized Hindsight returns a different task that the behavior is better suited for. Then, the behavior is relabeled with this new task before being used by an off-policy RL optimizer. Compared to standard relabeling techniques, Generalized Hindsight provides a substantially more efficient reuse of samples, which we empirically demonstrate on a suite of multi-task navigation and manipulation tasks. Videos and code can be accessed here: this https URL.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…the original goal of the policy (Andrychowicz, Wolski, Ray, Schneider, Fong, Welinder, McGrew, Tobin, Abbeel, and Zaremba, 2017; Rauber, Ummadisingu, Mutz, and Schmidhuber, 2017; Li, Pinto, and Abbeel, 2020; Kim, Lee, Kim, Ryu, Lee, and Zhang, 2021b; Moro, Likmeta, Prati, Restelli, et al., 2022).”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, unknown): “Many of these methods employ hindsight relabeling [6, 25, 62, 78], wherein experience is relabeled with an outcome that occurred in the future.”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, methodology): “Furthermore, our learning domain consists of tasks with randomized elements (e.g., objects spawn in random locations), making even one task in our work a multi-task learning problem by prior definitions [13].”
Hindsight Foresight Relabeling for Meta-Reinforcement Learning (2021, methodology): “Prior 44 relabeling methods for multi-task RL have used the total reward of the trajectory under different tasks 45 to guide the relabeling [11, 12].”