Feedback controller parameterizations for Reinforcement Learning

By John W. Roberts, Ian R. Manchester, Russ Tedrake. 2011

35 citations recorded by the source corpus.

Reinforcement Learning offers a very general framework for learning controllers, but its effectiveness is closely tied to the controller parameterization used. Especially when learning feedback controllers for weakly stable systems, ineffective parameterizations can result in unstable controllers and poor performance both in terms of learning convergence and in the cost of the resulting policy. In this paper we explore four linear controller parameterizations in the context of REINFORCE, applying them to the control of a reaching task with a linearized flexible manipulator. We find that some natural but naive parameterizations perform very poorly, while the Youla Parameterization (a popular parameterization from the controls literature) offers a number of robustness and performance advantages.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, background): “Roberts et al. (2011) demonstrate that care needs to be taken when selecting closed-loop parameterizations for weakly stable systems, and suggest forms that are particularly robust during learning.”
A Self-Tuning LQR Approach Demonstrated on an Inverted Pendulum (2014, methodology): “Directly tuning the weights of an LQR cost to improve control performance is also considered by Roberts et al. [2011], who discuss different parameterizations of feedback controllers in the context of reinforcement learning (one parameterization being through LQR weights). The authors consider a ful”
A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems (2017, background): “similar spirit, later work has considered constraining policy search to the space of stabilizing controllers [11].”
Guaranteed Safe Online Learning via Reachability: tracking a ground target using a quadrotor (2012, background): “Most machine learning techniques offer performance guarantees that are asymptotic in nature [4], [5]; very little research has been done in terms of quantifying their stability or robustness during the actual learning process (see [6] for an exception).”
Reducing Conservativeness in Safety Guarantees by Learning Disturbances Online: Iterated Guaranteed Safe Online Learning (2012, background): “One reason for this, as described by Roberts et al. [13], is that as they are being learned, natural parameterizations of controllers can behave poorly on systems near the edge of stability.”