Iterative Linear Quadratic Regulator Design for Nonlinear Biological Movement Systems

By Weiwei Li, E. Todorov. 2004

768 citations recorded by the source corpus.

This paper presents an Iterative Linear Quadratic Regulator (ILQR) method for locally-optimal feedback control of nonlinear dynamical systems. The method is applied to a musculo-skeletal arm model with 10 state dimensions and 6 controls, and is used to compute energy-optimal reaching movements. Numerical comparisons with three existing methods demonstrate that the new method converges substantially faster and finds slightly better solutions.

Subfields: Biomedical Engineering, Cognitive Neuroscience, Orthopedics and Sports Medicine, Neuroscience, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Cognitive Neuroscience

In Orthopedics and Sports Medicine

In Neuroscience

In Robotics & Control

Citations in the corpus

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, methodology): “To perform optimal control in the latent space of different models, we employ two trajectory optimization algorithms: iterative linear quadratic regulation (iLQR) [13] (for the plane and inverted pendulum) and approximate inference control (AICO) [14] (all other experiments).”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “…it will be useful to also deﬁne the value function V π ( x t , u t ) and advantage function A π ( x t , u t ) of a given policy π : (Deisenroth et al., 2013; Kober & Peters, 2012), two are particularly relevant for this work: iterative LQG (iLQG) (Li & Todorov, 2004) and Dyna-Q (Sutton, 1990).”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “sampling with MPC may be insufficient, and investigating other methods [39] in future work could improve performance.”
Deep spatial autoencoders for visuomotor learning (2015, methodology): “When the cost function is quadratic and the dynamics are linear-Gaussian, the optimal time-varying linear-Gaussian controller of the form p(ut|xt) = N (Ktxt + kt,Ct) can be obtained by using the LQR method.”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “rk [12] that uses human operators to implement such macro-actions for such simulation to real transfer, we implement these macro-actions using an iterative linearquadratic regulator (iLQR) controller [41,55]. iLQR leverages known system dynamics to output a dynamically feasible local reference traje”
Collective robot reinforcement learning with distributed asynchronous guided policy search (2016, methodology): “a) LQR with local models: To take a model-based approach to optimization of time-varying linear-Gaussian local policies, we can observe that, under time-varying linearGaussian dynamics, the local policies can be optimized analytically using the LQR method [24].”
Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning (2017, methodology): “…policies through a trajectory-centric reinforcement learning algorithm that alternates between ﬁtting local time-varying linear dynamics models, and updating the time-varying linear-Gaussian policies using the iterative linear-quadratic Gaussian regulator algorithm (iLQG) (Li & Todorov, 2004).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “In contrast, MBRL methods [45, 15, 72] can acquire dynamics models that may be utilized to perform unseen tasks at test time.”
Agnostic System Identification for Model-Based Reinforcement Learning (2012, neutral): “Non-linear systems with non-quadratic cost functions can also be solved approximately (local optima) using efﬁcient iterative linearization techniques such as iLQR(Li & Todorov, 2004).”
Exploring Model-based Planning with Policy Networks (2019, methodology): “GPS uses iLQG [25, 42, 40] as the local controller, and distill the knowledge into a policy neural network.”
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors (2015, methodology): “We will make use of the iterative linear quadratic regulator (iLQR) algorithm [16] to optimize the actions with respect to the cost under an estimated dynamics model.”
Learning Dexterous Manipulation Policies from Experience and Imitation (2016, methodology): “This type of iterative approach can be thought of as a variant of iterative LQR (Li and Todorov 2004), where the dynamics are ﬁtted to data.”
Control-Aware Representations for Model-based Reinforcement Learning (2020, methodology): “n that accounts for the curvature of the latent dynamics. This term regularizes the representation towards smoother latent dynamics, which are suitable for the locally-linear controllers, e.g., iLQR [Li and Todorov, 2004], used by these methods. In this paper, we take a few steps towards the above t”
Improved Learning of Dynamics Models for Control (2016, methodology): “(6) By using this form of cost function, along with a linearization of the learned dynamics model, we can formulate the policy synthesis problem as that of a Linear Quadratic Regulator, which allows the policy to be computed in closed-form.”
Optimal control with learned local models: Application to dexterous manipulation (2016, methodology): “The standard iterative LQR method can change the controller drastically at each iteration, which can cause it to visit parts of the state space where the fitted dynamics are arbitrarily incorrect, leading to divergence.”