Exploring parameter space in reinforcement learning

By Thomas Rückstieß, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, J. Urgen Schmidhuber. 2010

110 citations recorded by the source corpus.

This paper discusses parameter-based exploration methods for reinforcement learning. Parameter-based methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Parameter-based exploration unifies reinforcement learning and black-box optimization, and has several advantages over action perturbation. We review two recent parameter-exploring algorithms: Natural Evolution Strategies and Policy Gradients with Parameter-Based Exploration. Both outperform state-of-the-art algorithms in several complex high-dimensional tasks commonly found in robot control. Furthermore, we describe how a novel exploration method, State-Dependent Exploration, can modify existing algorithms to mimic exploration in parameter space.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, methodology): “We present it within the general MDP formulation of Section 4.1 for simplicity, although our experiments apply it to the greedy MDP of Section 4.2.”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “10The original work of (Rückstieß et al., 2010) considers an episode-based method where the policy parameter is sampled only at the start of an episode.”
Path Integral Policy Improvement with Covariance Matrix Adaptation (2012, methodology): “…for using CEM / CMAES for policy improvement is based on its empirical performance (Busoniu et al., 2011; Heidrich-Meisner and Igel, 2008; R¨uckstiess et al., 2010) (e.g. it is shown to outperform a particular gradient-based method), the PI 2 derivation (Theodorou et al., 2010)…”
Model-based contextual policy search for data-efficient generalization of robot skills (2017, methodology): “While the early PG algorithms were developed for action-based exploration, the ideas have been adopted to parameter-based exploration algorithms [43, 35, 33] as well.”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “The easiest method of including the noise is to include a fixed amount of noise [153].”
A Survey on Policy Search for Robotics (2013, methodology): “However, estimating the full covariance matrix can also be difficult [69] if the parameter space becomes high-dimensional (|Θ| > 50) as the covariance matrix has O(|Θ|2) elements.”
Multi-Dimensional Deep Memory Atari-Go Players for Parameter Exploring Policy Gradients (2010, methodology): “We therefore train our Go-playing networks using the novel Policy Gradients with Parameter-based Exploration (PGPE, [8]), which have recently been shown to be very successful at optimizing the parameters of large Neural Network controllers [9].”