Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings

By John D. Co-Reyes, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, Sergey Levine. 2018

159 citations recorded by the source corpus.

In this work, we take a representation learning perspective on hierarchical reinforcement learning, where the problem of learning lower layers in a hierarchy is transformed into the problem of learning trajectory-level generative models. We show that we can learn continuous latent representations of trajectories, which are effective in solving temporally extended and multi-stage problems. Our proposed model, SeCTAR, draws inspiration from variational autoencoders, and learns latent representations of trajectories. A key component of this method is to learn both a latent-conditioned policy and a latent-conditioned model which are consistent with each other. Given the same latent, the policy generates a trajectory which should match the trajectory predicted by the model. This model provides a built-in prediction mechanism, by predicting the outcome of closed loop policy behavior. We propose a novel algorithm for performing hierarchical RL with this model, combining model-based planning in the learned latent space with an unsupervised exploration objective. We show that our model is effective at reasoning over long horizons with sparse rewards for several simulated tasks, outperforming standard reinforcement learning methods and prior methods for hierarchical reasoning, model-based planning, and exploration.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “, 2011); and other areas such as hierarchical RL (Peng et al., 2017; Co-Reyes et al., 2018; Jiang et al., 2019).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “, 2015), especially for a system (Co-Reyes et al., 2018) like HRL due to its complexity and lack of clear supervision.”
Shaping Belief States with Generative Environment Models for RL (2019, background): “But evidence of the effect of overshooting has been restricted to the agent’s performance evaluation [33, 31].”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, background): “Prior works leverage imitation [26, 29] and reinforcement [7] within this framework.”
Generalization to New Actions in Reinforcement Learning (2020, methodology): “We accomplish this by incorporating insights from trajectory autoencoders (Wang et al., 2017; Co-Reyes et al., 2018) in HVAE.”
Federated Deep Reinforcement Learning (2019, unknown): “…(Yang et al., 2019)), we propose a novel federated learning framework based on reinforcement learning (Sutton & Barto, 1998; Mnih et al., 2015; Co-Reyes et al., 2018), i.e., Fed erated deep R einforcement L earning ( FedRL ), which aims to learn a private Q-network policy for each agent by…”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, background): “Co-Reyes et al. (2018) learn a latent space of low-level skills that can be controlled by a higher-level policy, framed within the setting of hierarchical RL.”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “However, it also figures heavily in offline RL, where it is used to estimate conditional distributions over actions that serve to constrain the learned policy to avoid out-of-distribution behavior that is not supported under the dataset [16, 31, 17]; imitation learning, where it is used to fit an ex”
Shared Autonomy with Learned Latent Actions (2020, background): “There is also extensive work in learning latent representations for components in RL [4, 6, 31, 20, 9].”
Ready Policy One: World Building Through Active Learning (2020, background): “This would require behavioral representations, and some notion of distance in behavioral space (Co-Reyes et al., 2018), and may lead to increased sample efficiency as we could better target specific state action pairs.”
Deep Hierarchical Planning from Pixels (2022, methodology): “…et al., 2021), discovering latent skills via mutual-information maximization (Gregor et al., 2016; Florensa et al., 2017; Hausman et al., 2018; Eysenbach et al., 2018; Achiam et al., 2018; Merel et al., 2018; Laskin et al., 2022; Sharma et al., 2019; Xie et al., 2020; Hafner et al., 2020b;…”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, methodology): “We use the same sets of hyperparameters as in Co-Reyes et al. (2018).”
Learning latent actions to control assistive robots (2021, unknown): “Robots can learn low-dimensional models of states [40], dynamics [49,50], movement primitives [39], trajectories [14], plans [32], policies [17], skills [42], and action representations for reinforcement learning [11].”
Federated Reinforcement Learning (2019, methodology): “Different from previous federated learning, we propose a novel federated learning framework based on reinforcement learning (Sutton & Barto, 1998; Mnih et al., 2015; Co-Reyes et al., 2018), i.”
Discovering Motor Programs by Recomposing Demonstrations (2020, background): “SeCTAR (Co-Reyes et al., 2018) builds a latent variable conditioned policy and model that are constrained to be consistent with one another, and uses the learned policies and model for hierarchical reinforcement learning.”
Efficient Exploration via State Marginal Matching (2019, methodology): “We further decompose the SMM objective into a mixture of distributions, and derive an algorithm for learning a mixture of policies that resembles the mutual-information objectives in recent work (Achiam et al., 2018; Eysenbach et al., 2018; Co-Reyes et al., 2018).”
Dynamics-aware Embeddings (2019, background): “Unlike this work, both SeCTAR and HIRO learn state-dependent low-level policies, not action representations.”
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation (2019, methodology): “[11] use a policy decoder to generate sequence of actions and a state decoder to predict the dynamics given the same latent feature as the input.”
On Reward-Free Reinforcement Learning with Linear Function Approximation (2020, unknown): “…Bellemare et al., 2016, Houthooft et al., 2016, Tang et al., 2017, Florensa et al., 2017, Pathak et al., 2017, Tang et al., 2017, Achiam et al., 2017, Hazan et al., 2018, Burda et al., 2018, Colas et al., 2018, Co-Reyes et al., 2018, Nair et al., 2018, Eysenbach et al., 2018, Pong et al., 2019].”
Controlling Assistive Robots with Learned Latent Actions (2019, background): “Recent research has learned latent dynamics [17], trajectories [18],”