Planning with Goal-Conditioned Policies

By Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, Sergey Levine. 2019

255 citations recorded by the source corpus.

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, such that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, such that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based tasks that require non-greedy, multi-staged behavior.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reus”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Many prior methods choose a representation learning objectives that reconstruct the input state [31, 46, 48, 49, 70, 91, 93, 141] while others use contrastive representation learning methods [74, 89, 95, 111, 117].”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, unknown): “Non-parametric methods for exploration and planning with images have been considered (Warde-Farley et al., 2019; Nasiriany et al., 2019; Wang et al., 2019; Liu et al., 2020; Shah et al., 2021) and could also beneﬁt from improved representation learning.”
Goal-Directed Planning via Hindsight Experience Replay (2022, background): “In such context, the use of sub-goals (intermediate states between the current state and the goal state) have been studied to accelerate learning and generalize over the state-space (Nasiriany et al., 2019; Jurgenson et al., 2020).”