Prediction and Control with Temporal Segment Models

By Nikhil Mishra, Pieter Abbeel, Igor Mordatch. 2017

66 citations recorded by the source corpus.

We introduce a method for learning the dynamics of complex nonlinear systems based on deep generative models over temporal segments of states and actions. Unlike dynamics models that operate over individual discrete timesteps, we learn the distribution over future state trajectories conditioned on past state, past action, and planned future action trajectories, as well as a latent prior over action trajectories. Our approach is based on convolutional autoregressive models and variational autoencoders. It makes stable and accurate predictions over long horizons for complex, stochastic systems, effectively expressing uncertainty and modeling the effects of collisions, sensory noise, and action delays. The learned dynamics model and action prior can be used for end-to-end, fully differentiable trajectory optimization and model-based policy optimization, which we use to evaluate the performance and sample-efficiency of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Value Prediction Network (2017, methodology): “In cases where the observation-prediction model is differentiable with respect to continuous actions, backpropagation can be used to find the optimal action [19] or to compute value gradients [11].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, background): “[10] proposed a relatively complex time-convolutional model for dynamics prediction, but only demonstrated results on low-dimensional (2D) manipulation tasks.”
Meta-Learning with Temporal Convolutions (2017, methodology): “…by the use of similar architectures for image generation (van den Oord et al. [28]) and modeling stochastic dynamical systems (Mishra et al. [17]), these models based on temporal convolutions (TC) are not only expressive and versatile, but lend themselves well to problems with an underlying…”
Model-Ensemble Trust-Region Policy Optimization (2018, background): “Mishra et al. (2017) learn a latent variable dynamic model over temporally extended segments of the trajectory, and train a policy using gradient-based optimization over the latent space.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, neutral): “First, differentiable trajectory optimization methods propagate the gradients of the policy or value function through the learned dynamics model [31, 32] .”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, neutral): “This type of prediction is similar to the prediction made by prior work on multi-step models (Mishra et al., 2017; Venkatraman et al., 2016): predict the state after τ actions.”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, methodology): “Mishra et al. (2017) uses trajectory segment models for planning but has no mechanism for exploration and does not consider hierarchical tasks.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “These methods require an accurate and reliable model and will typically suffer from modeling bias, hence these models are still limited to short horizon prediction in more complex domains [27, 30, 13, 42].”
Model Based Planning with Energy Based Models (2019, background): “2 Related Work A number of model classes have been explored for model-based planning in robotics and artificial intelligence literature, from feed-forward, recurrent [11], temporal segment [12] and Bayesian [13] neural networks, to locally-linear models [14, 15] and Gaussian processes [16].”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “ch [3, 33]. These methods require an accurate and reliable model and will typically suffer from modeling bias, hence these models are still limited to short horizon prediction in more complex domains [27, 30, 13, 42]. Another class of model-based methods rely only on local system models to compute t”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Mishra et al. (2017) learns action priors used to generate actions for MPC that embed past actions in a latent space for constrained planning, similarly to how we skill constraint, but only considers a fairly simple manipulation task.”
PLAS: Latent Action Space for Offline Reinforcement Learning (2020, methodology): “[19] samples action sequences from the CVAE when they perform trajectory optimization with the learned latent dynamics model.”
End-to-End Differentiable Physics for Learning and Control (2018, methodology): “…et al. [2017]), often with a specific loss function or network structured aimed at predicting on the relevant time scales [Abbeel and Ng, 2005, Mishra et al., 2017], or the model used is a more “pure” physics model, where learning essentially corresponds to traditional system identification…”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, methodology): “These methods require an reliable model and will typically suffer from modeling bias, hence these models are still limited to short horizon prediction in more complex domains (Mishra et al., 2017).”
Deep Dynamical Modeling and Control of Unsteady Fluid Flows (2018, background): “In the machine learning community, recent work focused on learning deep dynamical models from data has showed that these models can enable more sample-efficient learning of effective controllers [37]–[39].”
Learning Stable Deep Dynamics Models (2020, background): “Owing to their representational power, neural networks have long been a natural choice for modeling the function f [7, 14, 13, 6].”