Synthesis and stabilization of complex behaviors through online trajectory optimization

By Yuval Tassa, Tom Erez, E. Todorov. 2012

973 citations recorded by the source corpus.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, neutral): “For the physical control problems we compare our results to a baseline computed by a planner [8] that has full access to the underlying simulated dynamics and its derivatives (see supplementary information).”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “They include a variety of obstacle courses for agents with different bodies ( Quadruped , Planar Walker , and Humanoid [5, 6]).”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “, 2015b), Simple Humanoid (Tassa et al., 2012; Schulman et al., 2015b), and Full Humanoid (Tassa et al.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “Planning is a natural and powerful approach to decision making problems with known dynamics, such as game playing and simulated robot control (Tassa et al., 2012; Silver et al., 2017; Morav ˇ cík et al., 2017).”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “The iLQG algorithm optimizes trajectories by iteratively constructing locally optimal linear feedback controllers under a local linearization of the dynamics ˆ p ( x t +1 | x t , u t ) = N ( f x t x t + f u t u t , F t ) and a quadratic expansion of the rewards r ( x t , u t ) (Tassa et al., 2012).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “22: for each (si,ai), compute Â(si,ai) = Qφk+1(si,ai)− Ea∼πk,g(a|si)[Qφk+1(si,a)] 23: ∇θk,gJ(πθk,g ) ≈ 1N∇θk,g log πθk,g (si,ai)Â(si,ai) 24: θk,g+1 ← θk,g + απ∇θk,gJ(πθk,g ) 25: end for 26: θk + 1← θk,Gπ . update policy parameters 27: end for utilize it for planning at test time, often by means of”
Generalization in anti-causal learning (2018, background): “Tassa et al. (2012) solve a set of MuJoCo tasks in real time without any learning , by online planning over the simulator itself.”
Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, methodology): “We use a variant of DDP called iterative LQR (iLQR), which requires only a first order expansion of the dynamics [29].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “MBRL enables the use of generic priors like smoothness and physics [22] for model learning, and a wide variety of planning algorithms [23, 24, 25, 26, 27].”
Deep visual foresight for planning robot motion (2016, background): “The high dimensionality of image observations presents a substantial challenge to model-based approaches, which have been most successful for low-dimensional non-visual tasks [21] such as helicopter control [22], locomotion [23], and robotic cutting [24].”
DeepMind Control Suite (2018, background): “Humanoid (54, 21, 67): A simplified humanoid with 21 joints, based on the model in (Tassa et al., 2012).”
Towards Generalization and Simplicity in Continuous Control (2017, methodology): “The complexity of systems solvable with deepRL methods is not yet at the level of what can be achieved with trajectory optimization (planning) in simulation [5, 6, 7], or with hand-crafted controllers on physical robots (e.”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Iterative LQR (iLQR) is often employed to con- 108 Behavioral Cloning trol a system of which the dynamics is not accurately known [Todorov and Li, 2005, Abbeel et al., 2010, Tassa et al., 2012]. iLQR learns a linear feedback controller to follow a trajectory through an iterative learning process.”
Neural probabilistic motor primitives for humanoid control (2018, methodology): “These can be obtained by sample-based optimization (e.g. Ding et al. (2015)) or by differential dynamic programming (Morimoto & Atkeson, 2003; Tassa et al., 2012; 2014).”
Exploring Model-based Planning with Policy Networks (2019, methodology): “GPS uses iLQG [25, 42, 40] as the local controller, and distill the knowledge into a policy neural network.”
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors (2015, background): “suited for MPC because it is fast, can be effectively used with short horizons, and readily allows warm-starting with the previous solution [18].”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, methodology): “Approaches such as cost shaping [36] can also be interpreted as hand specifying an approximate value function, and has been successfully employed with MPC [37].”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “…with the following dynamic programming updates, which proceed from the last time step t = T to the ﬁrst step t = 1 : It can be shown (e.g., in [39]) that the action a t that minimizes the second-order approximation of the Q-function at every time step t is given by This action is a linear…”