Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task

By Stephen James, Andrew J. Davison, Edward Johns. 2017

291 citations recorded by the source corpus.

End-to-end control for robot manipulation and grasping is emerging as an attractive alternative to traditional pipelined approaches. However, end-to-end methods tend to either be slow to train, exhibit little or no generalisability, or lack the ability to accomplish long-horizon or multi-stage tasks. In this paper, we show how two simple techniques can lead to end-to-end (image to velocity) execution of a multi-stage task, which is analogous to a simple tidying routine, without having seen a single real image. This involves locating, reaching for, and grasping a cube, then locating a basket and dropping the cube inside. To achieve this, robot trajectories are computed in a simulator, to collect a series of control velocities which accomplish the task. Then, a CNN is trained to map observed images to velocities, using domain randomisation to enable generalisation to real world images. Results show that we are able to successfully accomplish the task in the real world with the ability to generalise to novel environments, including those with dynamic lighting conditions, distractor objects, and moving objects, including the basket itself. We believe our approach to be simple, highly scalable, and capable of learning long-horizon tasks that have until now not been shown with the state-of-the-art in end-to-end robot control.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Driving Policy Transfer via Modularity and Abstraction (2018, background): “n [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color images is known to simplify transfer [50,28,27]. When dealing with color images, domain randomization [46,21,42,43] enables direct sim-to-real generalization by maximizing the diversity of textures and ”
A Framework for Data-Driven Robotics (2019, background): “Sim2real: The sim2real framework aims to seamlessly transfer policies trained in simulated environments to the real world [29]–[33].”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “Recently there has been a large amount of work in learning control policies in simulation and then transferring these to the real world [50]–[55].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Another direction which has shown promising results is using sim-to-real transfer [14], [22]–[24], but our interest in a small amount of incremental human effort per new task is challenging for these methods, since they currently require significant engineering effort for each new simulation scenari”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “This approach has been employed for manipulation [24], visual servoing [25], locomotion [26], and navigation [23].”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “To sidestep the constraints of training on real hardware we embrace the sim2real paradigm which has recently shown promising results [17, 39, 47].”
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018, background): “e RGB cameras, there is considerable value in studying grasping systems that solely use monocular RGB images. A number of recent works have also examined using randomized simulated environments [13], [14] for simulationto-real world transfer for grasping and grasping-like manipulation tasks, extendi”
Learning Latent Plans from Play (2019, neutral): “s were more stability and oscillations were reduced. Lastly, our work is related to prior research on few-shot learning of skills from demonstrations (Finn et al. [15], Wang et al. [64], James et al. [22], Alet et al. [2], Duan et al. [11]). While our method does not require demonstrations to perfor”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, methodology): “Such approaches have the potential to be general enough to learn a wide range of tasks, and they have been shown to be capable of performing tasks that traditional methods in robotics have found difficult, such as when close and complicated coordination is required between vision and control [2], or”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “Domain randomization [51, 61, 25, 38, 3, 24] is another technique that is particularly popular in robotics, where an agent is trained on a wide range of variations of sensory inputs, with the intention that this forces the input processing layers of the network to extract semantically relevant featu”
Learning ambidextrous robot grasping policies (2019, background): “This simulation-to-reality transfer problem is a long-standing challenge in robot learning ( 14 – 17 ).”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “Sim2real aims to seamlessly transfer policies trained in simulated environments to the real world [30, 67, 55, 5, 11, 32].”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, unknown): “Deep learning is the only approach so far that has been able to deal with end-to-end control [5, 19, 48].”
Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies (2016, unknown): “…learning context: (i) the use of simulators, simulated and synthetic data [Bateux et al. , 2017; D’Innocente et al. , 2017; Tobin et al. , 2017; James et al. , 2017]; (ii) methods that transfer the learned models to real-world scenarios [Fitzgerald et al. , 2015; Tzeng et al. , 2016]; (iii)…”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “…et al., 2017; Levine et al., 2015; Kalashnikov et al., 2018; Lange et al., 2012; OpenAI et al., 2018; Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including in goal-conditioned settings (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017;…”
FutureMapping: The Computational Structure of Spatial AI Systems (2018, background): “[21]), or via reinforcement based on the the achievement of discrete tasks such as object placement or local navigation.”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, background): “Domain randomization has proved successful at detecting colored geometric shapes on a table [7], flying a quadcopter indoors [8], or learning visuomotor control for reaching [9] or pick-and-place [10] of a brightly colored cube.”