Neuroevolutionary reinforcement learning for generalized control of simulated helicopters

By Rogier Koppejan, Shimon Whiteson. 2011

56 citations recorded by the source corpus.

This article presents an extended case study in the application of neuroevolution to generalized simulated helicopter hovering, an important challenge problem for reinforcement learning. While neuroevolution is well suited to coping with the domain’s complex transition dynamics and high-dimensional state and action spaces, the need to explore efficiently and learn on-line poses unusual challenges. We propose and evaluate several methods for three increasingly challenging variations of the task, including the method that won first place in the 2008 Reinforcement Learning Competition. The results demonstrate that (1) neuroevolution can be effective for complex on-line reinforcement learning tasks such as generalized helicopter hovering, (2) neuroevolution excels at finding effective helicopter hovering policies but not at learning helicopter models, (3) due to the difficulty of learning reliable models, model-based approaches to helicopter hovering are feasible only when domain expertise is available to aid the design of a suitable model representation and (4) recent advances in efficient resampling can enable neuroevolution to tackle more aggressively generalized reinforcement learning tasks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

A comprehensive survey on safe reinforcement learning (2015, methodology): “However, in some situations in which the safety of the agent is particularly important, for example in expensive robotic platforms, researchers are paying increasing attention not only to the long-term reward maximization, but also to damage avoidance (Mihatsch and Neuneier, 2002; Hans et al., 2008;”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “A Neuroevolutionary RL method [140] Simulation Experiments Helicopter control [19]”
Phase-Parametric Policies for Reinforcement Learning in Cyclic Environments (2018, background): “Koppejan and Whiteson (2011) propose a neuro-evolutionary method based approach for the generalized helicopter hovering problem.”
OFFER: Off-Environment Reinforcement Learning (2017, methodology): “2014) as well as to the generalised helicopter hovering task (Koppejan and Whiteson 2011), which is known to be characterised by significant rare events.”
Safe Exploration of State and Action Spaces in Reinforcement Learning (2012, methodology): “This represents a clear advantage over other approaches, e.g., evolutionary RL (Mart´ın H. & Lope, 2009; Koppejan & Whiteson, 2011) where the extraction of knowledge about the known space by the agent is impossible using the weights of the neural-networks.”
Evolutionary Bilevel Optimization for Complex Control Tasks (2015, methodology): “ForOl, a specialized NE algorithm that has additional mutation replacement and crossover averaging operators [15] is used.”