The limits and potentials of deep learning for robotics

By Niko Sünderhauf, Oliver Brock, Walter J. Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford and 1 more. 2018

566 citations recorded by the source corpus.

The application of deep learning in robotics leads to very specific problems and research questions that are typically not addressed by the computer vision and machine learning communities. In this paper we discuss a number of robotics-specific learning, reasoning, and embodiment challenges for deep learning. We explain the need for better evaluation metrics, highlight the importance and unique challenges for deep robotic learning in simulation, and explore the spectrum between purely data-driven and model-driven approaches. We hope this paper provides a motivating overview of important research directions to overcome the current limitations, and helps to fulfill the promising potentials of deep learning in robotics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “In order to fully integrate deep learning into robotics, learning systems should reliably estimate the uncertainty in their predictions [1].”
Continual Learning for Robotics (2019, background): “Learning, embodiment, and reasoning are presented as the three great families of challenges for robotics in (Sünderhauf et al., 2018).”
Mean Field Analysis of Neural Networks: A Law of Large Numbers (2018, background): “[51], [22], and [40] describe applications in robotics.”
Zero-Shot Anticipation for Instructional Activities (2018, background): “This best reﬂects the situation under which service robots will be introduced [18, 49].”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, background): “, 2018), and the dynamics of ocean dynamics and turbulent flows (Aksamit et al., 2019; Sünderhauf et al., 2018; Brunton et al., 2020).”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “The seminal survey by Kober et al. (15) was written before the deep learning era, and the general deep learning for robotics survey by Sunderhauf et al. (16) was written when DRL accomplishments were primarily in simulation.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “S¨underhauf et al. (2018) identiﬁed current areas of research in deep learning that were relevant to robotics, and described a few challenges in applying deep learning techniques to robotics.”
Mean Field Analysis of Deep Neural Networks (2019, background): “Neural networks also find increasing more applications in engineering, robotics, medicine, and finance (see [21], [22], [43], [15], [32], [1], [36], [37], [38], and [39]).”
Classification-Reconstruction Learning for Open-Set Recognition (2018, background): “Closed-set classifiers are error-prone to samples of unknown classes, and this limits their usability [46, 43].”
Probabilistic Object Detection: Definition and Evaluation (2018, background): “For safe and trusted operation in robots or autonomous systems, CNNs must express meaningful uncertainty information [1, 14, 15, 19, 43, 49].”
Robot Programming without Coding (2020, methodology): “Both [16] and [17] point out a list of missing pieces that can be thought of as growth opportunities for today’s robotics, programming and AI.”
Pitfalls in Machine Learning Research: Reexamining the Development Cycle (2020, unknown): “Machine learning can look to the ﬁeld of robotics for best practices, as it has always done this and never hesitates to point out the mismatch between datasets and the real world [46].”
Reinforcement Learning Control of a Forestry Crane Manipulator (2021, background): “nt advances in machine learning (ML) in general, and reinforcement learning (RL) in particular, have inspired progress in the development of intelligent systems in the context of robotic manipulation [1,2]. These tasks often require multiple-skill acquisition in high-dimensional and dynamically comp”
Residual Reinforcement Learning for Robot Control (2018, methodology): “In this sense, our method is related to differentiable programming [29] or automated algorithm design [30].”
Fine-Grained Continual Learning (2019, background): “However, this approach is rather impractical when learning from high-dimensional streaming data, especially in highly constrained computational platforms and embedded systems [6, 7].”
Adversarial Robustness with Semi-Infinite Constrained Learning (2021, background): “Learning is at the core of many modern information systems, with wide-ranging applications in clinical research [1–4], smart grids [5–7], and robotics [8–10].”
A survey of uncertainty in deep neural networks (2021, background): “differ from those often addressed in computer vision and other off-line settings [299].”