Reinforcement and Imitation Learning for Diverse Visuomotor Skills

By Yuke Zhu, Ziyun Wang, Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas and 1 more. 2018

347 citations recorded by the source corpus.

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in this https URL

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, background): “Zhu et al. (2018) combine imitation learning and RL by summing an RL loss and a generative adversarial loss from imitating the demonstrator (Ho and Ermon, 2016).”
Solving Rubik's Cube with a Robot Hand (2019, background): “policy that transfers to similar environments but with different dynamics. Domain randomization was also used to plan robust grasps [65, 66, 107] and to transfer learned locomotion [105] and grasping [123] policies for relatively simple robots. Pinto et al. [83] propose to use adversarial training t”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Learning : Several recent works (Veˇcer´ık et al., 2017; Nair et al., 2017; Gao et al., 2018; Zhu et al., 2018) have introduced algorithms that unify reinforcement learning and imitation learning such that the learning agent can beneﬁt from both expert demonstrations and interactions with the…”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Using imitation learning as a way to bootstrap RL has been previously leveraged by a number of deep RL algorithms [28, 29, 30], where a ﬂat imitation learning initialization is improved using reinforcement learning with additional auxiliary objectives.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Other than directly using BC with RL, imitation learning has been combined with RL in a variety of manners, such as mixed with adversarial methods [Zhu et al., 2018, Kang et al., 2018], used for pre-training [Pfeiffer et al., 2018], modifying the replay buffer [Večerík et al., 2017, Gulcehre et…”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “By contrast, IRL generally requires fewer expert samples, but it relies on a prohibitive amount of agent interaction with the environment [16, 39, 42] due to the need to do reinforcement learning with a learned reward function.”
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning (2020, methodology): “In particular, the SpaceMouse from 3Dconnexion is a 3D mouse that we used extensively for collecting demonstrations [17, 10] and debugging task designs.”
Visual Imitation Made Easy (2020, background): “Prominent works in Visual Imitation Learning (VIL) have demonstrated utility in intricate manipulation skills such as pushing, grasping, and stacking [5, 7].”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “Imitation and Reinforcement Learning from Suboptimal Demonstrations : Recent work has tried to leverage off-policy deep reinforcement learning in conjunction with a set of demonstrations to account for suboptimal data and learn policies that outperform the demonstrations [12, 14, 28, 36, 38].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Another direction which has shown promising results is using sim-to-real transfer [14], [22]–[24], but our interest in a small amount of incremental human effort per new task is challenging for these methods, since they currently require significant engineering effort for each new simulation scenari”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “…supervision can be provided through contemporary game interfaces such as a keyboard and mouse [26, 27], a video game controller [28], a 3D-mouse [29, 30], special purpose master-slave interfaces[31, 32], or through free-space positioning interfaces such as virtual reality (VR) [33, 34, 35].”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “community [17], [16], [18], [19], [1], [20], [21] but these methods have not been applied to real world manipulation with multi-fingered hands.”
Learning dexterous in-hand manipulation (2018, methodology): “Domain randomization was also used to plan robust grasps [38, 39, 63] and to transfer learned locomotion [61] and grasping [69] policies for relatively simple robots.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, background): “There has been a growing interest in combining extrinsic sparse reward signals with imitation learning for guided exploration (Zhu et al., 2018; Kang et al., 2018; Le et al., 2018; Vecer´ık et al., 2017).”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, background): “In many cases however, it is either too challenging to optimize a given reward (e.g. due to sparsity of signal), or it is simply impossible to design a reward function that captures the intricate details of desired outcomes [3, 4, 5, 6, 7, 8].”
Task-Relevant Adversarial Imitation Learning (2019, background): “Although many variants are introduced in the literature [Li et al., 2017, Fu et al., 2018, Merel et al., 2017, Zhu et al., 2018, Baram et al., 2017], making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.”
Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations (2018, methodology): “Using a combination of IL and RL to obtain a sample efficient and robust learning algorithm has previously been explored in robotics in the context of manipulation tasks [27], [28].”
Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation (2018, methodology): “ to imitate [1]. That can become dicult, though, when the agent is a physical robot. That is why more and more studies use synthetic data to train imitation learning methods in a virtual environment [2,3,4,5]. Project webpage:https://daphneantotsiou.github.io/task-oriented-retargeting.html arXiv:181”