Autonomous helicopter control using reinforcement learning policy search methods

By J. Bagnell, Jeff Schneider, Andrew Bagnell. 2001

334 citations recorded by the source corpus.

Many control problems in the robotics field can be cast as partially observed Markovian decision problems (POMDPs), an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers suffice for good sub-optimal control, and recent research in the artificial intelligence community has focused on policy search methods as techniques for finding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximum-likelihood Markovian model. We consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider briefly other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To validate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by flying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain.

Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “…have been routinely applied in robotics, in part because they straightforwardly handle continuous and high-dimensional action spaces [3] and applications include manipulation [26, 13, 25, 37, 18, 5, 39, 8], locomotion e.g. [16, 21], and a range of other challenges such as helicopter ﬂight [1].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Approaches have included using Gaussian process models for controlling blimps (Ko et al., 2007), and using linear regression (Koppejan and Whiteson, 2009) and locally-weighted Bayesian regression (Bagnell and Schneider, 2001) for helicopter control.”
Reinforcement learning in robotics: A survey (2013, methodology): “, 2010), and general stochastic optimization methods (such as Nelder–Mead (Bagnell and Schneider, 2001), cross-entropy (Rubinstein and Kroese, 2004) and population-based methods (Goldberg, 1989)) to approaches coming from optimal control such as DDP (Atkeson, 1998)”
Dual Policy Iteration (2018, methodology): “Previous work suggests a policy that optimizes all the training models simultaneously is stable and robust during testing [24, 25], as the training environments together act as “regularization" to avoid overfitting and provide generalization.”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “…the most impressive robotic learning applications so far were achieved using the simplest possible model parameterization, namely linear models (Bagnell & Schneider, 2001; Abbeel et al., 2006; Levine & Abbeel, 2014; Watter et al., 2015; Levine et al., 2016a; Kumar et al., 2016), where the…”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Impressive results with model-based RL have been obtained using simple linear models [16, 17, 18, 19].”
Humanoid robots learning to walk faster: from the real world to simulation and back (2013, background): “Machine learning is an enticing alternative to manual generation of these skills and behaviors, and indeed there have been some notable successes of such [4, 15, 20].”
Learning concurrent motor skills in versatile solution spaces (2012, background): “REPS is an information theoretic policy search approach which is based on the insight that the information loss between two policy steps should be bounded [19], [20], [8].”
Space-indexed dynamic programming: learning to follow trajectories (2008, methodology): “such as those described in ( Bagnell & Schneider, 2001; Ng et al., 2004).”
Autonomous Helicopter Aerobatics through Apprenticeship Learning (2010, background): “However, when τ is known, the optimization problem in Equation (12) greatly simplifies thanks to context specific independencies (Boutilier et al. 1996). When τ is fixed, we obtain a model such as that shown in Figure 3. In this model we can directly estimate the multinomial parameters d in closed f”
Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models (2015, methodology): “These ideas have been successfully applied to RL, control and robotics [2–7], for instance.”
A survey of robot learning from demonstration (2009, positive): “The opposite approach derives a model of robot dynamics from demonstration but then uses the model in simulation, for example to simulate robot statewhen employing RL to optimize a NN controller for autonomous helicopter flight [78] and inverted helicopter hovering [6].”
Transfer learning for reinforcement learning on a physical robot (2010, background): “However, a few notable successes notwithstanding (e.g., flying RC helicopters [3] [9] and quadruped walking [8]), RL algorithms have typically been applied only in simulation, or at best trained in simulation with the eventual learned behaviors run on a real robot (e.g., [6], [10], and [5]).”
Probabilistic model-based imitation learning (2013, methodology): “For instance, model-free methods based on sampling trajectories directly from the robot [30,40] or model-based RL algorithms that learn forward models of the robot and, subsequently, use them for predictions [5,14,15,27] are suitable.”