Learning omnidirectional path following using dimensionality reduction

By J. Kolter, Andrew Y. Ng. 2007

33 citations recorded by the source corpus.

We consider the task of omnidirectional path following for a quadruped robot: moving a four-legged robot along any arbitrary path while turning in any arbitrary manner. Learning a controller capable of such motion requires learning the parameters of a very high-dimensional policy, a difficult task on a real robot. Although learning such a policy can be much easier in a model (or “simulator”) of the system, it can be extremely difficult to build a sufficiently accurate simulator. In this paper we propose a method that uses a (possibly inaccurate) simulator to identify a low-dimensional subspace of policies that spans the variations in model dynamics. This subspace will be robust to variations in the model, and can be learned on the real system using much less data than would be required to learn a policy in the original class. In our approach, we sample several models from a distribution over the kinematic and dynamics parameters of the simulator, then formulate an optimization problem that can be solved using the Reduced Rank Regression (RRR) algorithm to construct a low-dimensional class of policies that spans the major axes of variation in the space of controllers. We present a successful application of this technique to the task of omnidirectional path following, and demonstrate improvement over a number of alternative methods, including a hand-tuned controller. We present, to the best of our knowledge, the first controller capable of omnidirectional path following with parameters optimized simultaneously for all directions of motion and turning rates.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “It is often faster to fine-tune a controller learned in simulation than to learn from scratch in the real world [9], [21].”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, methodology): “Latent space models for RL have been explored in several prior works, though not in the context of meta-learning or learning exploration strategies (Hausman et al., 2018; Flo-Meta-Reinforcement rensa et al., 2017; Kolter & Ng, 2007).”
Value Function Approximation in Reinforcement Learning Using the Fourier Basis (2011, methodology): “In this section we describe the Fourier series for one and multiple variables, and use it to define the univariate and multivariate Fourier bases.”
AMP (2021, unknown): “These models specify controls through a learned latent representation, which is then mapped to controls for the underlying system [Burgard et al. 2008; Florensa et al. 2017; Hausman et al. 2018; Heess et al. 2016].”
On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient (2010, background): “They have been applied to a variety of complex real-world reinforcement learning problems, such as hitting a baseball with an articulated arm robot [1], constrained humanoid robotic motion planning [2], and learning gaits for legged robots [3, 4, 5].”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, unknown): “actions) of the underlying system [22].”