Estimating the Maximum Expected Value in Continuous Reinforcement Learning Problems

By Carlo D'Eramo, Alessandro Nuara, Matteo Pirotta, Marcello Restelli. 2017

20 citations recorded by the source corpus.

This paper is about the estimation of the maximum expected value of an infinite set of random variables.This estimation problem is relevant in many fields, like the Reinforcement Learning (RL) one.In RL it is well known that, in some stochastic environments, a bias in the estimation error can increase step-by-step the approximation error leading to large overestimates of the true action values. Recently, some approaches have been proposed to reduce such bias in order to get better action-value estimates, but are limited to finite problems.In this paper, we leverage on the recently proposed weighted estimator and on Gaussian process regression to derive a new method that is able to natively handle infinitely many random variables.We show how these techniques can be used to face both continuous state and continuous actions RL problems.To evaluate the effectiveness of the proposed approach we perform empirical comparisons with related approaches.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, background): “It is known as the max-operator bias in statistics (D’Eramo et al., 2017) and as the ”winner’s curse” in economics (Smith & Winkler, 2006; Thaler, 2012).”
Sequential Test for the Lowest Mean: From Thompson to Murphy Sampling (2018, background): “The problem of estimating μ∗ = mina μa was studied in [34] and subsequently [7, 31, 8].”
Interleaved Q-Learning with Partially Coupled Training Process (2019, methodology): “The extension of GE to an infinite number of RVs is reported in [5].”