Learning To Reach Goals Without Reinforcement Learning

By Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, Sergey Levine. 2019

38 citations recorded by the source corpus.

Imitation learning algorithms provide a simple and straightforward approach for training control policies via supervised learning. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator to provide the demonstrations. In this paper, we ask: can we take insights from imitation learning to design algorithms that can effectively acquire optimal policies from scratch without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, when tasks correspond to different goals, every trajectory is a successful demonstration for the goal state that it actually reaches. We propose a simple algorithm for learning goal-reaching behaviors without any demonstrations, complicated user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning with demonstrations, we show how this approach can effectively learn goal-reaching policies from scratch. We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems, while yielding advantages in terms of stability and use of offline data.

Subfields: Artificial Intelligence, Computer Science Applications, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “[42] extends prior UDRL methods to use state goal conditioning, rather than rewards, and Paster et al.”
Reward-Conditioned Policies (2019, methodology): “The central idea behind our method – that suboptimal trajectories can serve as optimal supervision for other tasks or problems – has recently been explored for goal -conditioned policies, both with reinforcement learning [1, 16, 34] and supervised learning [8].”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “Ghosh et al. [2019] studied an algorithm that is similar to the one by Lynch et al. [2020] in using pure SL and conditioning actions on fixed goal states, and similar to ours in that all data for learning is generated by the agent itself.”
Grounding Language in Play (2020, methodology): “…technique trains goal conditioned policies to reach any previously visited state upon demand, with many recent examples in RL (Nair et al., 2018; Pong et al., 2018; Gupta et al., 2019; Pong et al., 2019; Ghosh et al., 2019; Eysenbach et al., 2020) and IL (Lynch et al., 2019; Ding et al., 2019).”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Prior methods for goal-conditioned supervised learning (Oh et al., 2018; Ding et al., 2019; Ghosh et al., 2019; Sun et al., 2019; Lynch et al., 2019) have a similar motivation: learning “useful” information for control without access to rewards.”