Optimal and Autonomous Control Using Reinforcement Learning: A Survey

By Bahare Kiumarsi, Kyriakos G. Vamvoudakis, Hamidreza Modares, Frank L. Lewis. IEEE Transactions on Neural Networks and Learning Systems, 2018

807 citations recorded by the source corpus.

This paper reviews the current state of the art on reinforcement learning (RL)-based feedback control solutions to optimal regulation and tracking of single and multiagent systems. Existing RL solutions to both optimal <inline-formula> <tex-math notation="LaTeX">$\mathcal {H}_{2}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$\mathcal {H}_\infty $ </tex-math></inline-formula> control problems, as well as graphical games, will be reviewed. RL methods learn the solution to optimal control and game problems online and using measured data along the system trajectories. We discuss Q-learning and the integral RL algorithm as core algorithms for discrete-time (DT) and continuous-time (CT) systems, respectively. Moreover, we discuss a new direction of off-policy RL for both CT and DT systems. Finally, we review several applications.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/399dbf5ba3d00f37b8476bef04ad0dee90f85102

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qinglai Wei
2. Derong Liu
3. Denis Yarats
4. Biao Luo
5. Bo Zhao
6. Shan Xue
7. Thien Huynh-The
8. Angela P. Schoellig

In Computational Theory and Mathematics

1. Derong Liu
2. Qinglai Wei
3. Biao Luo
4. Bo Zhao
5. Shan Xue
6. Denis Yarats
7. Angela P. Schoellig
8. Ming Liu

In Control and Systems Engineering

1. Derong Liu
2. Qinglai Wei
3. Biao Luo
4. Denis Yarats
5. Angela P. Schoellig
6. Bo Zhao
7. Ming Liu
8. Zhu Han

Citations in the corpus

• Adaptive Dynamic Programming for Control: A Survey and Recent Advances (2021, background): “It is noted that most of the existing ADP literature is biased toward discrete-time systems [62], [63], [69], [101], [102].”
• On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “…with an unknown environment has wide-reaching applications in, e.g. , robotics (Kober et al., 2013; Polydoros and Nalpantidis, 2017), control (Lillicrap et al., 2015; Kiumarsi et al., 2017), finance (Fischer, 2018), and gaming (Vinyals et al., 2019; Berner et al., 2019; Justesen et al., 2019).”
• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “Previous review works focused on specific techniques—for example, learning-based model predictive control (MPC) (7), iterative learning control (ILC) (8, 9), model-based RL (10), data-efficient policy search (11), imitation learning (12), or the use of RL in robotics (13, 14) and in optimal control ”
• Artificial Intelligence for the Metaverse: A Survey (2022, background): “in which an agent learns to attain a goal in an uncertain and complex environment [9].”
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, methodology): “To make the robot learn to navigate, we adopt a reinforcement learning perspective, which is built on recent success of deep RL algorithms for solving challenging control tasks [12-15].”