A Tour of Reinforcement Learning: The View from Continuous Control

By B. Recht. 2018

747 citations recorded by the source corpus.

This article surveys reinforcement learning from the perspective of optimization and control, with a focus on continuous control applications. It reviews the general formulation, terminology, and typical experimental implementations of reinforcement learning as well as competing solution paradigms. In order to compare the relative merits of various techniques, it presents a case study of the linear quadratic regulator (LQR) with unknown dynamics, perhaps the simplest and best-studied problem in optimal control. It also describes how merging techniques from learning theory and control can provide nonasymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. The article concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and control might be combined to approach these challenges.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Electrical and Electronic Engineering

Citations in the corpus

Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “Previous review works focused on specific techniques—for example, learning-based model predictive control (MPC) (7), iterative learning control (ILC) (8, 9), model-based RL (10), data-efficient policy search (11), imitation learning (12), or the use of RL in robotics (13, 14) and in optimal control ”
Combining Optimal Control and Learning for Visual Navigation in Novel Environments (2019, background): “Furthermore, learning such policies is extremely sample inefﬁcient requiring millions of samples to acquire desired behavior [36].”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “The main challenge is DRL’s notorious training instability with a high variance as discussed in [11, 29, 1, 56], which makes it difﬁcult to correctly rank the sampled networks within a limited search time.”
Data-Enabled Predictive Control: In the Shallows of the DeePC (2018, methodology): “approaches usually require a large number of data samples to perform well, and are often sensitive to hyperparameters leading to non-reproducible and highly variable outcomes [4].”
Learning-Based Model Predictive Control: Toward Safe Learning in Control (2020, background): “While input constraints typically optimizing over a sequence of control laws { k }, which can make use of all information in the and approximate dynamic programming or reinforcement learning (16, 17).”
Data-Driven Model Predictive Control With Stability and Robustness Guarantees (2019, background): “, stability or constraint satisfaction of system variables [1], [2].”