Learning Actionable Representations with Goal-Conditioned Policies

By Dibya Ghosh, Abhishek Gupta, Sergey Levine. 2018

119 citations recorded by the source corpus.

Representation learning is a central challenge across a range of machine learning areas. In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems. Most prior work on representation learning has focused on generative approaches, learning representations that capture all underlying factors of variation in the observation space in a more disentangled or well-ordered manner. In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable." These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, without explicit reconstruction of the observation. We show how these representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks. We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Our trajectory-collecting agent uses both random walks and a simultaneously learned curiosity-driven goalconditioned policy (Ghosh et al., 2018; Nair et al., 2018).”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering…”
Goal-Directed Planning via Hindsight Experience Replay (2022, background): “In this line of work, various works study the problem of compact representations of multiple tasks (Dhiman et al., 2018; Ghosh et al., 2018).”
Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, background): “Ghosh et al. (2018) estimate distance between two states based on the actions a pre-trained policy would take to reach them.”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “2D Room Navigation (Ghosh et al., 2019) This environment requires an agent to navigate to points in an environment with four rooms that connect to adjacent rooms.”
Generalization in Generation: A closer look at Exposure Bias (2019, methodology): “Similar formulations of expected reward have recently been proposed as goal-conditioned policies (Ghosh et al., 2018).”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, methodology): “[17] observe that one can learn such a metric with access to a goal-conditioned policy by optimizing for a goal metric that reflects the number of actions needed to reach a particular goal.”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, background): “Some work focuses on representation learning speciﬁcally for the goal-conditioned setting — Ghosh et al. (2019) uses goal-conditioned policies to extract a representation where Euclidean distances between states correspond to expected differences between actions to reach them.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “By conditioning on some piece of information, such as a goal, reward function parameterization, or reward value, such experience can be used for simple behavior cloning [5, 6, 8, 10, 17, 23, 26, 29, 32, 34, 38].”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “…including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016; Tessler et al., 2017) and integrating goal-conditioned behaviors (Ghosh et al., 2019; Nachum et al., 2019).”
GFlowNet Foundations (2021, unknown): “In practice, a GFlowNet will never be perfectly trained to completion, and we should think of such an outcome-conditioned GFlowNet similarly to a goal-conditioned policy in RL (Ghosh et al., 2018) or the rewardconditioned upside-down RL (Schmidhuber, 2019).”