Efficient robust policy optimization

By Christopher G. Atkeson. 2012

12 citations recorded by the source corpus.

We provide efficient algorithms to calculate first and second order gradients of the cost of a control law with respect to its parameters, to speed up policy optimization. We achieve robustness by simultaneously designing one control law for multiple models with potentially different model structures, which represent model uncertainty and unmodeled dynamics. Providing explicit examples of possible unmodeled dynamics during the control design process is easier for the designer and is more effective than providing simulated perturbations to increase robustness, as is currently done in machine learning. Our approach supports the design of deterministic nonlinear and time varying controllers for both deterministic and stochastic nonlinear and time varying systems, including policies with internal state such as observers or other state estimators. We highlight the benefit of control laws made up of collections of simple policies where only one component policy is active at a time. Controller optimization and learning is particularly fast and effective in this situation because derivatives are decoupled.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “Also in the limit of deterministic policies and models, the recursions we have derived in Algorithm 1 reduce to those of [2].”
Dual Policy Iteration (2018, methodology): “Previous work suggests a policy that optimizes all the training models simultaneously is stable and robust during testing [7, 4], as the training environments together act as “regularization" to avoid overﬁtting and provide generalization.”