Combining the benefits of function approximation and trajectory optimization

By Igor Mordatch, E. Todorov. 2014

124 citations recorded by the source corpus.

Neural networks have recently solved many hard problems in Machine Learning, but their impact in control remains limited. Trajectory optimization has recently solved many hard problems in robotic control, but using it online remains challenging. Here we leverage the high-fidelity solutions obtained by trajectory optimization to speed up the training of neural network controllers. The two learning problems are coupled using the Alternating Direction Method of Multipliers (ADMM). This coupling enables the trajectory optimizer to act as a teacher, gradually guiding the network towards better solutions. We develop a new trajectory optimizer based on inverse contact dynamics, and provide not only the trajectories but also the feedback gains as training data to the network. The method is illustrated on rolling, reaching, swimming and walking tasks.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “for deterministic policies under known dynamics [28].”
Learning contact-rich manipulation skills with guided policy search (2015, methodology): “Previous guided policy search algorithms have also focused on leveraging known models of the environment [19], [20], [21] and more sophisticated trajectory optimization algorithms [22].”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “An alternative set of approaches [38, 39, 40, 41, 42] explore using local trajectory optimization to generate a dataset for training a global policy through imitation learning.”
Guided Policy Search as Approximate Mirror Descent (2016, background): “For linear-Gaussian p i ( u t | x t ) and (nonlinear) conditionally Gaussian π θ ( u where µ π and Σ π can be any function (such as a deep neural network), the KL-divergence D KL ( π θ ( u t | x t,i,j ) (cid:107) p i ( u t | x t,i,j )) can easily be evaluated and differentiated in closed form [6].”
Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators (2019, background): “Furthermore, it could be combined with the function approximation abilities of neural networks to learn and represent these policies [18], [19].”
Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor Observations (2015, background): “[27, 38]) as well as playing Go from raw visual input [32] and learning attention policies for object recognition [36].”
Learning Complex Neural Network Policies with Trajectory Optimization (2014, background): “Concurrently with our work, Mordatch and Todorov (2014) proposed another trajectory optimization algorithm for guiding policy search.”