Sim-to-Real Robot Learning from Pixels with Progressive Nets

By Andrei A. Rusu, Matej Vecerík, Thomas Rothörl, Nicolas Manfred Otto Heess, Razvan Pascanu, Raia Hadsell. 2016

568 citations recorded by the source corpus.

Applying end-to-end learning to solve complex, interactive, pixel-driven control tasks on a robot is an unsolved problem. Deep Reinforcement Learning algorithms are too slow to achieve performance on a real robot, but their potential has been demonstrated in simulated environments. We propose using progressive networks to bridge the reality gap and transfer learned policies from simulation to the real world. The progressive net approach is a general framework that enables reuse of everything from low-level visual features to high-level policies for transfer to new tasks, enabling a compositional, yet simple, approach to building complex skills. We present an early demonstration of this approach with a number of experiments in the domain of robot manipulation that focus on bridging the reality gap. Unlike other proposed approaches, our real-world experiments demonstrate successful task learning from raw visual input on a fully actuated robot manipulator. Moreover, rather than relying on model-based trajectory optimisation, the task learning is accomplished using only deep reinforcement learning and sparse rewards.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Rusu et al. [36] explore using the progressive network architecture to adapt a model that is pre-trained on simulated pixels, and ﬁnd it has better sample efﬁciency than ﬁne-tuning or training in the real-world alone.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “The third category of strategies, dy-2 namic network architectures , maintains past knowledge thanks to architectural modiﬁcations while learning [Rusu et al., 2016a, Fernando et al., 2017, Li and Hoiem, 2018, Fernando et al., 2017].”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Moreover, sim2real transfer is of major interest in the robotic learning community, because of limited data and supervision in the real world (Tobin et al., 2017; Rusu et al., 2017; Peng et al., 2018; James et al., 2019; Yan et al., 2020; Andrychowicz et al., 2020).”
Relational Deep Reinforcement Learning (2018, background): “Recent advances in deep reinforcement learning (deep RL) [1, 2, 3] are in part driven by a capacity to learn good internal representations to inform an agent’s policy.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Rusu et al. (2017) propose to progressively adapt the learned deep features and representations from the synthetic domain to the real-world domain.”
Progress & Compress: A scalable framework for continual learning (2018, methodology): “Progressive networks (Rusu et al., 2016a) are a method within this category designed for continual learning.”
Continual Learning with Adaptive Weights (CLAW) (2019, background): “This is one way to enable multi-task transfer whilst mitigating catastrophic forgetting, which has proven to be effective (Rusu et al., 2016b; Fernando et al., 2017; Yoon et al., 2018), albeit with limitations.”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “on. A number of works employ specialized learning techniques and network architectures to facilitate transfer: for example, variants of domain adaptation [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color images is known to simplify transfer [50,28,27]. W”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “resentations of real and synthetic data [16], using generative models to make synthetic images more realistic [17], fine-tuning the policies trained in simulation in the real world [18], learning inverse dynamics models [19], multiobjective optimization of task fitness and transferability to the rea”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “For example, a policy can be pre-trained in simulation and then fine-tuned on robots [41].”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, neutral): “In contrast to prior work on domain adaptation [35, 41], our method does not require any real images during training.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “ Although model-free RL is typically too sample inefﬁ- cient, some recent works have been able to reduce sample complexity through careful task set-up, and apply them on real world tasks. Rusu et. al [31] learn a policy in simulation and transfer it to the real world. Other work [9] shows that by pr”
StarCraft II: A New Challenge for Reinforcement Learning (2017, methodology): “These techniques have also proven successful in reinforcement learning problems, yielding significant successes in Atari [20], the game of Go [31], three-dimensional virtual environments [3] and simulated robotics domains [17, 28].”
Learning dexterous in-hand manipulation (2018, background): “Domain adaption methods [66, 21], progressive nets [53], and learning inverse dynamics models [10] were all proposed to help with sim to real transfer.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, methodology): “…et al., 2015; Mnih et al., 2016; Levine & Abbeel, 2014; Heess et al., 2015; Lillicrap et al., 2015; Schulman et al., 2016) has not yet translated as well to reality, despite various attempts (Tobin et al., 2017; Tzeng et al., 2016; Daftry et al., 2016; Finn et al., 2015; Rusu et al., 2016).”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, background): “[40] learn simple visuomotor policies for a Jaco robot arm and transfer to reality using progressive networks [39].”