Robust Locally-Linear Controllable Embedding

By Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Hai Bui, A. Ghodsi. 2017

101 citations recorded by the source corpus.

Embed-to-control (E2C) is a model for solving high-dimensional optimal control problems by combining variational auto-encoders with locally-optimal controllers. However, the E2C model suffers from two major drawbacks: 1) its objective function does not correspond to the likelihood of the data sequence and 2) the variational encoder used for embedding typically has large variational approximation error, especially when there is noise in the system dynamics. In this paper, we present a new model for learning robust locally-linear controllable embedding (RCE). Our model directly estimates the predictive conditional density of the future observation given the current one, while introducing the bottleneck between the current and future observations. Although the bottleneck provides a natural embedding candidate for control, our RCE model introduces additional specific structures in the generative graphical model so that the model dynamics can be robustly linearized. We also propose a principled variational approximation of the embedding posterior that takes the future observation into account, and thus, makes the variational approximation more robust against the noise. Experimental results show that RCE outperforms the E2C model, and does so significantly when the underlying dynamics is noisy.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “E2C (Watter et al., 2015) and RCE (Banijamali et al., 2017) embed images into a latent space, where they learn local-linear latent transitions and plan for actions using LQR.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Control with latent dynamics E2C (Watter et al., 2015) and RCE (Banijamali et al., 2017) embed images to predict forward in a compact space to solve simple tasks.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “As mentioned in Section 6, our E2C-like ablation was unable to make progress for the 2D navigation task, though we were able to get more successful results by ﬁxing the position of the goal to the bottom right as is done in the image-based 2D navigation task considered in E2C [43] and RCE [4].”
Block Contextual MDPs for Continual Learning (2021, background): “Some prior works train a single ﬁxed dynamics model but introduce additional constraints that ensure the latent dynamics are locally linear (Watter et al., 2015; Banijamali et al., 2018) or that the learned dynamics model is invariant to translation dynamics (Fragkiadaki et al., 2015).”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…in the pixel space (Finn and Levine, 2017; Ebert et al., 2018) or in a latent space (Levine et al., 2016; Finn et al., 2016; Watter et al., 2015; Banijamali et al., 2018; Zhang et al., 2019; Hafner et al., 2019, 2020; Ha and Schmidhuber, 2018; Kipf et al., 2019; Chen et al., 2020) and can…”
Learning Semantic Embedding Spaces for Slicing Vegetables (2019, methodology): “Both, E2C [8] and RCE [9] map input images into a latent embedding space, where they learn locally linear transition models and plan for actions using LQR.”
Control-Aware Representations for Model-based Reinforcement Learning (2020, methodology): “Although CARL allows us to learn a reward function in the latent space, similar to several other LCE works [Watter et al., 2015, Banijamali et al., 2018, Levine et al., 2020, Shu et al., 2020], in this paper, we assume that a compatible reward function is given.”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “Our 2-dimensional navigation task is similar to Watter et al. (2015) and Banijamali et al. (2018) where an agent controls its velocity in a bounded planar system to reach a specified target.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “This has been demonstrated by learning dynamics using future predictions (Watter et al., 2015; Oh et al., 2017; Ebert et al., 2017; Banijamali et al., 2018; Ha & Schmidhuber, 2018; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang et al., 2019), learning belief representations (”
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (2019, methodology): “This class that we refer to as learning controllable embedding (LCE) throughout the paper, include recently developed algorithms, such as E2C (Watter et al., 2015), RCE (Banijamali et al., 2018), and SOLAR (Zhang et al., 2019).”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, methodology): “More similar to our work, model-based reinforcement learning [5,10,7,18] planning in latent space typically assumes access to the low-dimensional states of the environment and plan the dynamics directly in continuous state space, which can be more eﬃcient compared to Bellman backups of traditional…”