3D Simulation for Robot Arm Control with Deep Q-Learning

By Stephen James, Edward Johns. 2016

111 citations recorded by the source corpus.

Recent trends in robot arm control have seen a shift towards end-to-end solutions, using deep reinforcement learning to learn a controller directly from raw sensor data, rather than relying on a hand-crafted, modular pipeline. However, the high dimensionality of the state space often means that it is impractical to generate sufficient training data with real-world experiments. As an alternative solution, we propose to learn a robot controller in simulation, with the potential of then transferring this to a real robot. Building upon the recent success of deep Q-networks, we present an approach which uses 3D simulations to train a 7-DOF robotic arm in a control task without any prior knowledge. The controller accepts images of the environment as its only input, and outputs motor actions for the task of locating and grasping a cube, over a range of initial configurations. To encourage efficient learning, a structured reward function is designed with intermediate rewards. We also present preliminary results in direct transfer of policies over to a real robot, without any further training.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [19], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer well to the real world [35].”
The limits and potentials of deep learning for robotics (2018, background): “Partial success on transferring from simulation to a real robot has been reported [8], [48], [134], [138].”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “2974707 there is now a large body of work looking at increasing the capabilities of robotic agents through the use of reinforcement learning [3], [4], meta-learning [5]–[7], multi-task learning [8],”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, methodology): “Unlike previous methods, which sought to bridge the reality gap with high ﬁdelity rendering [20], their systems used only low ﬁdelity rendering and modeled differences in visual appearance by randomizing scene properties such as lighting, textures, and camera placement.”
Asymmetric Actor Critic for Image-Based Robot Learning (2017, background): “One approach is to make the simulator as close to the real world as possible [23]–[25].”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, neutral): “James and Johns [13] demonstrated a simulation-trained manipulation controller learned from photo-realistic rendering data that show similar behaviors in the real-world.”
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018, background): “Previous work has shown that direct transfer was not possible [31], while others have shown that transfer only works after the agent has received additional training in the real world [32].”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “There have been advances in applying these techniques to both simulated robotic platforms [17, 5], and real world platforms [3], but fundamental challenges remain, such as sample inefficiency, slow convergence, and the algorithms sensitivity to hyperparameters.”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, background): “However, these solutions often learn their skills from scratch and need a large amount of training data [3, 4, 5].”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “However, it is often not possible to naively transfer such skills directly due to the visual and dynamics differences between the two domains [26].”
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (2018, methodology): “d cube and place it in a basket, and the network of Zhang et al. [35] was used to position a robot near a blue cube. Other work explores learning robotic policies from a high-ﬁdelity rendering engine [14], generating high-ﬁdelity synthetic data via a procedural approach [33], or training object clas”
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods (2018, methodology): “In particular, we test generalization to new objects in a cluttered environment where objects may be obscured and the environment dynamics are complex, in contrast to works such as [40], [34], and [19], which consider grasping simple geometric shapes such as blocks.”
Real-World Robot Learning with Masked Visual Pre-training (2022, background): “, joint velocities, end49 effector poses, etc) directly from observations [17, 18, 19], without the need to perform explicit 3D 50 scene understanding [20], grasp planning [21], and motion planning [22, 23].”
Virtual-to-Real: Learning to Control in Visual Semantic Segmentation (2018, background): “Most of the recent works on robot learning collect training data from simulators [ James and Johns, 2016; Rusu et al. , 2016; Sadeghi and Levine, 2017; Peng et al. , 2018; Tobin et al. , 2017; Zhu et al. , 2017].”