Reinforcement learning in robotics: A survey
By Jens Kober, J. Bagnell, Jan Peters. 2013
3,794 citations recorded by the source corpus.
Reinforcement learning offers to robotics a framework and set of tools for the design of sophisticated and hard-to-engineer behaviors. Conversely, the challenges of robotic problems provide both inspiration, impact, and validation for developments in reinforcement learning. The relationship between disciplines has sufficient promise to be likened to that between physics and mathematics. In this article, we attempt to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots. We highlight both key challenges in robot reinforcement learning as well as notable successes. We discuss how contributions tamed the complexity of the domain and study the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, a particular focus of our paper lies on the choice between model-based and model-free as well as between value-function-based and policy-search methods. By analyzing a simple problem in some detail we demonstrate how reinforcement learning approaches may be profitably applied, and we note throughout open questions and the tremendous potential for future research.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Computational Theory and Mathematics
- Sergey Levine
- Pieter Abbeel
- Ofir Nachum
- Chelsea Finn
- Shixiang Shane Gu
- Shimon Whiteson
- George Tucker
- Jan Peters
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- End-to-End Training of Deep Visuomotor Policies (2015, methodology): “Several recent papers provide surveys of policy search in robotics [3, 14].”
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “…that better algorithms and models are not enough; for human-centered applications such as healthcare (Yu et al., 2019), economics (Trott et al., 2021) and robotics (Kober et al., 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al., 2021; Hubinger et al., 2019).”
- Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Model-free RL has a myriad of applications in games [22, 37], robotics [16, 17], and marketing [18, 38], to name a few.”
- Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, background): “Neural networks are powerful function approximators, but they have significantly more parameters than the simpler linear models that are often used in robotic learning [23], [8].”
- Continuous Deep Q-Learning with Model-based Acceleration (2016, background): “…it will be useful to also define the value function V π ( x t , u t ) and advantage function A π ( x t , u t ) of a given policy π : (Deisenroth et al., 2013; Kober & Peters, 2012), two are particularly relevant for this work: iterative LQG (iLQG) (Li & Todorov, 2004) and Dyna-Q (Sutton, 1990).”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “, 2013), have been actively used in multiple domains, including robotic control (Kober et al., 2013), simulated autonomous navigation and self-driving cars (Sallab et al.”
- Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, background): “Reinforcement learning (RL) [18], [27] tackles control problems with nonlinear dynamics in a more general framework, which can be either model-based or model-free.”
- A Reinforcement Learning–Based Method for Management of Type 1 Diabetes: Exploratory Study (2019, neutral): “It has been applied successfully in different scientific fields such as robotics and control [29], manufacturing, and combinatorial search problems such as computer games [30,31].”
- Improving Robot Controller Transparency Through Autonomous Policy Explanation (2017, background): “Such control software can be complex to parse and interpret [21], particularly for modern robotics applications [20, 32].”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “ons of other moving agents. B. Deep Reinforcement Learning Robotics problems are often task-based. For a task-based problem with temporal structure, reinforcement learning is an appropriate approach (Kober et al., 2013). A variety of robotics control problems are solvable by reinforcement learning, ”
- Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, neutral): “much higher sample complexity [4], [12].”
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance (2017, neutral): “Reinforcement learning has been applied to a wide range of robotic problems, ranging from locomotion and manipulation to autonomous helicopter flight [14, 4].”
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “Learning good control strategies [21] automatically is an appealing alternative to tedious manual tuning.”
- End-to-End Driving Via Conditional Imitation Learning (2017, methodology): “In robotics, movement primitives have been used as building blocks for advanced motor skills [17], [26].”
- Consistent On-Line Off-Policy Evaluation (2017, methodology): “Reinforcement Learning (RL) techniques were successfully applied in fields such as robotics, games, marketing and more (Kober et al., 2013; Al-Rawi et al., 2015; Barrett et al., 2013).”
- Learning to Act by Predicting the Future (2016, neutral): “TD and policy gradient methods have since come to dominate the study of sensorimotor learning (Kober et al., 2013; Mnih et al., 2015; Sutton & Barto, 2017).”
- Benchmarks for Deep Off-Policy Evaluation (2021, background): “Reinforcement learning algorithms can acquire effective policies for a wide range of problems through active online interaction, such as in robotics (Kober et al., 2013), board games and video games (Tesauro, 1995; Mnih et al., 2013; Vinyals et al., 2019), and recommender systems (Aggarwal et al.,…”