Learning dexterous in-hand manipulation
By Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray and 6 more. 2018
2,231 citations recorded by the source corpus.
We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies that can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system such as friction coefficients and an object’s appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: https://youtu.be/jwSbzNHGflM.
Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Vikash Kumar
- Abhishek Gupta
- Abhinav Kumar Gupta
- Silvio Savarese
- Li Fei-Fei
In Mechanical Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Vikash Kumar
- Shixiang Shane Gu
- Igor Mordatch
- Aravind Rajeswaran
- Ofir Nachum
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Aviral Kumar
In Robotics & Control
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Silvio Savarese
- Abhinav Kumar Gupta
- Vikash Kumar
- Li Fei-Fei
- Dieter Fox
Citations in the corpus
- On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “(2018); Arjovsky et al. (2019); Krueger et al.”
- Solving Rubik's Cube with a Robot Hand (2019, methodology): “The block reorientation task was previously proposed in [84] and solved on a physical robot hand in [77].”
- On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “Model Parameter Values β-VAE β [1, 2, 4, 6, 8] AnnealedVAE cmax [5, 10, 25, 50, 75] iteration threshold 100000 γ 1000 FactorVAE γ [10, 20, 30, 40, 50] DIP-VAE-I λod [1, 2, 5, 10, 20] λd 10λod DIP-VAE-II λod [1, 2, 5, 10, 20] λd λod β-TCVAE β [1, 2, 4, 6, 8] Table 2: Hyperparameters of different meth”
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Recent advances in deep reinforcement learning (RL) have shown remarkable success on challenging tasks (Andrychowicz et al., 2018; Mnih et al., 2015; Silver et al., 2016).”
- Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “2016) and Robotics (OpenAI et al. 2018), and even real-world settings (Bellemare et al.”
- MOReL : Model-Based Offline Reinforcement Learning (2020, background): “pen simultaneously. However, a direct embodiment of this trial and error learning process often leads to slow and sample inefficient learning that is often feasible only in the presense of a simulator [9, 10]. Similar to progress in other fields of AI, the ability to effectively learn from large offlin”
- AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “…learning (RL) in conjunction with function approximation has proliferated in recent years, demonstrating successful applications in fields such as robotics (Andrychowicz et al., 2018; Nachum et al., 2019a), game playing (Mnih et al., 2013), and conversational systems (Gao et al., 2019).”
- RoboNet: Large-Scale Multi-Robot Learning (2019, background): “. A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1,33,34], transfer across different dynamics [35,36,37,2], transfer across robot morphologies with invariant feature spaces [38] and modularity [”
- A Framework for Data-Driven Robotics (2019, methodology): “These methods have achieved remarkable success in dexterous in-hand manipulation [32] and deformable object manipulation [34].”
- Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Recent progress in mastering the games of Go [50] and Poker [43, 10] and advances in robotic control [36, 44, 30] present additional supporting evidence for the enormous potential of deep RL.”
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “1 Introduction Reinforcement learning (RL) has recently demonstrated a number of successes in various domains, such as games [25], robotics [1], and conversational systems [11, 18].”
- Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “In [9], randomization of physical properties and object appearance is used to train a dexterous robotic hand to perform in-hand manipulation.”
- Exploration by Random Network Distillation (2018, background): “…in RL seem to suggest that solving the most challenging tasks (Silver et al., 2016; Zoph & Le, 2016; Horgan et al., 2018; Espeholt et al., 2018; OpenAI, 2018; OpenAI et al., 2018) requires processing large numbers of samples obtained from running many copies of the environment in parallel.”
- Regularized Behavior Value Estimation (2021, methodology): “This paradigm has attained considerable success on Atari (Mnih et al., 2015), Go (Silver et al., 2017), StarCraft II and Dota 2 (Vinyals et al., 2019; Berner et al., 2019), and robotics (Andrychowicz et al., 2020).”
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “One potential approach to enable a robot to perform these manipulation tasks is with deep reinforcement learning (RL), which has achieved many successes in recent years [4, 5, 6, 7, 8].”
- What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “For continuous control tasks, on-policy algorithms such as REINFORCE [2], TRPO [10], A3C [14], PPO [17] and off-policy algorithms such as DDPG [13] and SAC [21] have enabled successful applications such as quadrupedal locomotion [20], self-driving [30] or dexterous in-hand manipulation [20, 25, 32].”
- Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “In combination with advances in deep learning and computational resources, this formulation has led to dramatic results in acting from perception (Mnih et al., 2015), game playing (Silver et al., 2016), and robotics (OpenAI et al., 2018) among others.”
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “-learning, multi-task reinforcement learning, benchmarks 1Introduction While reinforcement learning (RL) has achieved some success in domains such as assembly [1], ping pong [2], in-hand manipulation [3], and hockey [4], state-of-the-art methods require substantially more experience than humans to a”
- Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Another direction which has shown promising results is using sim-to-real transfer [14], [22]–[24], but our interest in a small amount of incremental human effort per new task is challenging for these methods, since they currently require significant engineering effort for each new simulation scenari”
- Benchmarking Model-Based Reinforcement Learning (2019, background): “While model-free algorithms have achieved success in areas including robotics [30, 44, 20, 1], video-games [34, 33], and motion animation [38], their high sample complexity limits largely their application to simulated domains.”