Deep Dynamics Models for Learning Dexterous Manipulation

By Anusha Nagabandi, Kurt Konolige, Sergey Levine, Vikash Kumar. 2019

474 citations recorded by the source corpus.

Dexterous multi-fingered hands can provide robots with the ability to flexibly perform a wide range of manipulation skills. However, many of the more complex behaviors are also notoriously difficult to control: Performing in-hand object manipulation, executing finger gaits to move objects, and exhibiting precise fine motor skills such as writing, all require finely balancing contact forces, breaking and reestablishing contacts repeatedly, and maintaining control of unactuated objects. Learning-based techniques provide the appealing possibility of acquiring these skills directly from data, but current learning approaches either require large amounts of data and produce task-specific policies, or they have not yet been shown to scale up to more complex and realistic tasks requiring fine motor skills. In this work, we demonstrate that our method of online planning with deep dynamics models (PDDM) addresses both of these limitations; we show that improvements in learned dynamics models, together with improvements in online model-predictive control, can indeed enable efficient and effective learning of flexible contact-rich dexterous manipulation skills -- and that too, on a 24-DoF anthropomorphic hand in the real world, using just 4 hours of purely real-world data to learn to simultaneously coordinate multiple free-floating objects. Videos can be found at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “Surprisingly, actions depend more on past actions than they do on past states, reminiscent of the action smoothing used in some trajectory optimization algorithms (Nagabandi et al., 2019).”
Solving Rubik's Cube with a Robot Hand (2019, neutral): “In [71], the authors learn a forward dynamics model and use model predictive control to manipulate two Baoding balls with a Shadow hand.”
MOPO: Model-based Offline Policy Optimization (2020, background): “Uncertainty quantiﬁcation, a key ingredient to our approach, is critical to good performance in model-based RL both theoretically [62, 73, 43] and empirically [11, 6, 49, 38, 7], in optimal control [61, 2, 33].”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, methodology): “2 using the zeroth order stochastic optimizer from [28].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Alternative representations including keypoints [3, 4, 5, 6] and dense descriptors [9, 10, 11] have shown to be capable of category-level generalization and representing deformable objects, but still struggle to represent scenes with an unknown number of objects (e.”
Ready Policy One: World Building Through Active Learning (2020, background): “In recent times there has been great deal of progress in MBRL, with success in Atari games [30, 44], and dexterous manipulation [36], while progress has been made on continuous control benchmarks [31, 9, 10, 28].”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “Recent Model-based deep RL approaches [Hafner et al. 2018; Chua et al. 2018; Nagabandi et al. 2019], where the algorithm plans against a learned transition model of the environment, show a lot of promise for improving sample efficiency.”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “Surprisingly, actions depend more on past actions than they do on past states, reminiscent of the action smoothing used in some trajectory optimization algorithms [41].”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “In some cases, it is possible to execute a scripted behavior to reset the environment [28, 32, 47, 39, 46, 1].”
Masked Visual Pre-training for Motor Control (2022, background): “…hands but rely on explicit state estimation (Handa et al., 2020; Huang et al., 2021), expert policies (Chen et al., 2021a), human demonstrations (Ra-jeswaran et al., 2018; Radosavovic et al., 2021; Qin et al., 2021), human priors (Mandikal & Grauman, 2021), or models (Nagabandi et al., 2019).”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “…2018; Hafner et al., 2019b,a; Hansen et al., 2022, 2023; Hafner et al., 2023; Schrittwieser et al., 2020; Samsami et al., 2024), as well as real-world locomotion and manipulation tasks (Lee et al., 2020; Nagabandi et al., 2020; Finn et al., 2016; Ebert et al., 2017, 2018; Yen-Chen et al., 2020).”
A System for General In-Hand Object Re-Orientation (2021, methodology): “Model-based RL works learned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and used online optimal control to rotate a pen or Baoding balls on a Shadow hand.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recent Model-based deep RL approaches (Hafner et al. 2018; Chua et al. 2018; Nagabandi et al. 2019), where the algorithm plans against a learned transition model of the environment, show a lot of promise for improving sample efficiency.”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, unknown): “Some impressive RL-based systems for dexterous manipulation tackle a specific task with a specific object, like solving Rubik’s cube [1], shuffling Baoding balls [33], or reorienting”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “…additional components to enhance efficiency and stability, such as the use of model predictive control (Williams et al., 2017; Lowrey et al., 2019; Nagabandi et al., 2019), uncertainty quantification through Bayesian models (Deisenroth & Rasmussen, 2011), and ensembles of dynamics models…”
Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning (2020, methodology): “[21] learns deep dynamics modelsforavarietyofdifferentsimulateddexterousmanipulationtasks, usinggroundtruthobjectstates, and one task on real hardware, using an external camera-based 3D object position tracker.”