Combining model-based policy search with online model learning for control of physical humanoids

By Igor Mordatch, Nikhil Mishra, Clemens Eppner, Pieter Abbeel. 2016

57 citations recorded by the source corpus.

We present an automatic method for interactive control of physical humanoid robots based on high-level tasks that does not require manual specification of motion trajectories or specially-designed control policies. The method is based on the combination of a model-based policy that is trained off-line in simulation and sends high-level commands to a model-free controller that executes these commands on the physical robot. This low-level controller simultaneously learns and adapts a local model of dynamics on-line and computes optimal controls under the learned model. The high-level policy is trained using a combination of trajectory optimization and neural network learning, while considering physical limitations such as limited sensors and communication delays. The entire system runs in real-time on the robot's computer and uses only on-board sensors. We demonstrate successful policy execution on a range of tasks such as leaning, hand reaching, and robust balancing behaviors atop a tilting base on the physical robot and in simulation.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, background): “…and tractable training in the large data regime, and have the potential to represent more complex functions, including non- 1Code available https://github.com/kchua/handful-of-trials smooth dynamics that are often present in robotics [Fu et al., 2016, Mordatch et al., 2016, Nagabandi et al., 2017].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “However, the most efficient model-based algorithms have used relatively simple function approximators, such as Gaussian processes [5, 3, 17], time-varying linear models [22, 19, 27, 40], and mixtures of Gaussians [15].”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
Learning modular neural network policies for multi-task and multi-robot transfer (2016, background): “Deep reinforcement learning (RL) has been successful in multiple domains, including learning to play Atari games [1], simulated and real locomotion [2], [3] and robotic manipulation [4].”
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model (2016, neutral): “An alternative is to learn dynamics models in an on-line fashion, constantly adapting the model based on an incoming stream of observed states and actions [11] [28] [43] [22].”
Model Based Planning with Energy Based Models (2019, methodology): “Using recursive least squares [15] to infer inverse dynamics also leads to good performance of -4694.”
Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real (2019, background): “For this reason, we avoid fine-tuning in the real world [30, 31], train wholly in simulation, and perform a zero-shot transfer to the real world.”
Deep reinforcement learning for tensegrity robot locomotion (2016, methodology): “Robotic learning methods have previously produced successful policies for tasks such as locomotion for bipeds and quadrupeds [2], [3], [4], [5], [6], [7].”
Probabilistically safe policy transfer (2017, background): “A number of recent papers have explored how to transfer policies from one domain to another [5], [6], [7], [11], [20], [24], [28], [29].”
Policy transfer via modularity and reward guiding (2017, methodology): “Most of previous work for policy transfer involves policies that do not involve object interaction [31], [32], [33], [34].”
The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, background): “Other work has investigated learning from simulation and then incorporating those models into robotic control [16].”
Generalizing Skills with Semi-Supervised Reinforcement Learning (2016, unknown): “Transfer learning has been applied to RL domains for transferring information across environments (Mordatch et al., 2016; Tzeng et al., 2016), robots (Devin et al., 2016), and tasks (Konidaris & Barto, 2006; Stolle & Atkeson, 2007; Dragan et al., 2011; Parisotto et al., 2016; Rusu et al., 2016).”
Policy transfer via modularity (2017, methodology): “A similar approach has been used for reaching, leaning, balancing, and other tasks that do not involve object interaction [25], [26], [27], [28].”
Residual Policy Learning (2018, methodology): “Another set of approaches uses the learned dynamics model (or inverse dynamics model) to perform trajectory optimization or model predictive control [11, 12].”