Asymmetric Actor Critic for Image-Based Robot Learning

By Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel. 2017

514 citations recorded by the source corpus.

Deep reinforcement learning (RL) has proven a powerful technique in many sequential decision making domains. However, Robotics poses many challenges for RL, most notably training on a physical system can be expensive and dangerous, which has sparked significant interest in learning control policies using a physics simulator. While several recent works have shown promising results in transferring policies trained in simulation to the real world, they often do not fully utilize the advantage of working with a simulator. In this work, we exploit the full state observability in the simulator to train better policies which take as input only partial observations (RGBD images). We do this by employing an actor-critic training algorithm in which the critic is trained on full states while the actor (or policy) gets rendered images as input. We show experimentally on a range of simulated tasks that using these asymmetric inputs significantly improves performance. Finally, we combine this method with domain randomization and show real robot experiments for several tasks like picking, pushing, and moving a block. We achieve this simulation to real world transfer without training on any real world data.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, methodology): “Noisy observations were provided to the policy network while the value network had access to non-noisy observations (since the value network is not needed when rolling out the policy on the robot and can thus use privileged information, as described in [81]).”
Visual Imitation Made Easy (2020, background): “It has also been extensively used in context of RL, where domain randomization is proposed to transfer learning from simulation to real world [36, 37, 38].”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, background): “Several works [32, 33] show promise in this direction.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[59] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel.”
Learning dexterous in-hand manipulation (2018, methodology): “Asymmetric Actor-Critic exploits the fact that the value network can have access to information that is not available on the real robot system.”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “This technique has also been been proposed concurrently by Pinto et al. [34].”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “While prior works on vision-based deep reinforcement learning for robotics can efficiently learn a variety of behaviors such as grasping [33, 32, 24], pushing [1, 8, 10], navigation [30, 21], and other manipulation tasks [26, 23, 30], they each make assumptions that limit their applicability to trai”
Learning Latent Plans from Play (2019, background): “Learning powerful goalconditioned policies with reinforcement learning can produce policies with good long-horizon performance, but is difficult in terms of both the number of samples required and the need for extensive on-policy exploration [43, 42, 31, 12, 14, 29, 33, 30].”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, background): “However, beyond simple robotic picking tasks [1, 2, 3], there exist little support to this level of optimism.”
Learning to Manipulate Deformable Objects without Demonstrations (2019, methodology): “To transfer our policies to the real-robot, we use domain randomization (DR) [62, 45, 48] in the simulator along with using images of real cloths.”
Data center cooling using model-predictive control (2018, background): “Recent years have seen considerable research advances in reinforcement learning (RL), with algorithms achieving impressive performance on game playing and simple robotic tasks [24, 29, 27].”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “Many methods employ some level of rendering randomization [59], [60], [61], [9], [62], [63], including lighting conditions and textures.”
A Generalizable Approach to Learning Optimizers (2021, background): “One detail related to using PPO is that the value function V can be given non-robust features [39] in addition to the robust features given to the policy π .”
Guided Meta-Policy Search (2019, unknown): “The former has been explored before in the context of single-task RL (Levine et al., 2016; Pinto et al., 2017), while the latter has been studied for methods that learn exploration strategies for sparse rewards using dense rewards for meta-training (Gupta et al., 2018).”