Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

By Fangyi Zhang, Jürgen Leitner, Michael Milford, Peter Corke. 2016

42 citations recorded by the source corpus.

While deep learning has had significant successes in computer vision thanks to the abundance of visual data, collecting sufficiently large real-world datasets for robot learning can be costly. To increase the practicality of these techniques on real robots, we propose a modular deep reinforcement learning method capable of transferring models trained in simulation to a real-world robotic task. We introduce a bottleneck between perception and control, enabling the networks to be trained independently, but then merged and fine-tuned in an end-to-end manner to further improve hand-eye coordination. On a canonical, planar visually-guided robot reaching task a fine-tuned accuracy of 1.6 pixels is achieved, a significant improvement over naive transfer (17.5 pixels), showing the potential for more complicated and broader applications. Our method provides a technique for more efficient learning and transfer of visuo-motor policies for real robotic systems without relying entirely on large real-world robot datasets.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, background): “Authors have explored simulation [17], [18], [19] and image composition [20] to generate training data that transfers to physical scenarios.”
Sim-to-real Transfer of Visuo-motor Policies for Reaching in Clutter: Domain Randomization and Adaptation with Modular Networks (2017, background): “Studies of deep visuo-motor policies indicate that the convolutional layers focus on perception, i.e., extracting useful information from visual inputs, while the fully connected (FC) layers perform control [23].”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, methodology): “Other methods focus on transferring visuomotor skills from simulated to real robots [22], [26].”
A review of modularization techniques in artificial neural networks (2019, background): “In Zhang et al. (2016), a modular deep Q network is proposed to facilitate transferring of a learnt robotic control network from simulation to real environment.”