Control-Aware Representations for Model-based Reinforcement Learning

By Brandon Cui, Yinlam Chow, Mohammad Ghavamzadeh. 2020

16 citations recorded by the source corpus.

A major challenge in modern reinforcement learning (RL) is efficient control of dynamical systems from high-dimensional sensory observations. Learning controllable embedding (LCE) is a promising approach that addresses this challenge by embedding the observations into a lower-dimensional latent space, estimating the latent dynamics, and utilizing it to perform control in the latent space. Two important questions in this area are how to learn a representation that is amenable to the control problem at hand, and how to achieve an end-to-end framework for representation learning and control. In this paper, we take a few steps towards addressing these questions. We first formulate a LCE model to learn representations that are suitable to be used by a policy iteration style algorithm in the latent space. We call this model control-aware representation learning (CARL). We derive a loss function for CARL that has close connection to the prediction, consistency, and curvature (PCC) principle for representation learning. We derive three implementations of CARL. In the offline implementation, we replace the locally-linear control algorithm (e.g.,~iLQR) used by the existing LCE methods with a RL algorithm, namely model-based soft actor-critic, and show that it results in significant improvement. In online CARL, we interleave representation learning and control, and demonstrate further gain in performance. Finally, we propose value-guided CARL, a variation in which we optimize a weighted version of the CARL loss function, where the weights depend on the TD-error of the current policy. We evaluate the proposed algorithms by extensive experiments on benchmark tasks and compare them with several LCE baselines.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, state abstractions could also be formed on the basis of value equivalence (Grimm, Barreto, Singh, and Silver, 2020; Sokota, Ho, Ahmad, and Kolter, 2021; Cui, Chow, and Ghavamzadeh, 2021) or for improved planning (Curtis, Silver, Tenenbaum, Lozano-Pérez, and Kaelbling, 2022).”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Cui et al. [314] proposed a modiﬁed value-guided CARL that optimizes the weighted version of the CARL loss function whose weights depend on the TD-error of the current policy.”