Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control

By Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, Peter Corke. 2015

289 citations recorded by the source corpus.

This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real hardware and real observation in a naive approach failed, but experiments show that the network works when replacing camera images with synthetic images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Robust Physical-World Attacks on Deep Learning Models (2017, background): “Deep Neural Networks (DNNs) are being applied with great success to a variety of areas ranging from speech processing [6] to medical diagnostics [4] and are increasingly being used to control physical objects such as cars [14], UAVs [2], and robots [30].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “esult indicated that the cognitive ability of the robot was extended by deep reinforcement learning model for more complicated indoor environments in an efﬁcient online-learning process continuously. Zhang et al. (2015) used DQN to train a target reaching task with a three-joint robot manipulator. T”
The limits and potentials of deep learning for robotics (2018, background): “Partial success on transferring from simulation to a real robot has been reported [8], [48], [134], [138].”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “This conclusion is in line with other work [5, 6].”
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints (2015, background): “Directly applying models learned in simulation to the real world typically does not succeed [35], due to systematic discrepancies between real and simulated data.”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, neutral): “[5] trained a deep Q-network in simulation for a target reaching task.”
Composable Deep Reinforcement Learning for Robotic Manipulation (2018, background): “transfers them into real world [14], [15], [16], [17].”
Attention-Aware Deep Reinforcement Learning for Video Face Recognition (2017, methodology): “employed reinforcement learning for vision control in robotics [45].”
One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay (2017, background): “Recent contributions include continuous control for simulated robots [9], motor control for manipulation directly from pixels [10], and simulation-to-real transfer for manipulation [11, 12].”
Runtime Neural Pruning (2017, methodology): “[45] employed reinforcement learning for vision control in robotics.”
Differentially Private Data Generative Models (2018, background): “Advanced machine learning techniques, and in particular deep neural networks (DNNs), have been applied with great success to a variety of areas, including speech processing [35], medical diagnostics [17], image processing [16], and robotics [64].”
Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not (2016, methodology): “Generalized mobile robotics research I.”
Sim-to-real Transfer of Visuo-motor Policies for Reaching in Clutter: Domain Randomization and Adaptation with Modular Networks (2017, background): “Recently learning approaches to tackle this problem have been presented [2]–[5], however a consistent issue faced by most approaches is the reliance on large amounts of data to train these models.”
Towards Cognitive Exploration through Deep Reinforcement Learning for Mobile Robots (2016, background): “As in [23], the motion control of a Baxter robot motivated by deep reinforcement learning could make sense only with simulated semantic images but not raw images taken by real cameras.”
Robust Physical-World Attacks on Deep Learning Visual Classification (2018, background): “these successes, they are increasingly being used as part of control pipelines in physical systems such as cars [8, 17], UAVs [4, 24], and robots [40].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “Nonetheless, value function-based methods such as Deep Q-Networks [264] and extensions thereof have been applied successfully to robotics tasks [435, 186].”
Learning Real-World Robot Policies by Dreaming (2018, background): “[13] showed that policy learning of a robot is possible with synthetic images but found that the learned policy does not work well when directly applied to real-world images.”