Smoothed Action Value Functions for Learning Gaussian Policies

By Ofir Nachum, Mohammad Norouzi, George Tucker, Dale Schuurmans. 2018

34 citations recorded by the source corpus.

State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value. We show that such smoothed Q-values still satisfy a Bellman equation, making them learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships, we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator. The approach is additionally amenable to proximal optimization by augmenting the objective with a penalty on KL-divergence from a previous policy. We find that the ability to learn both a mean and covariance during training leads to significantly improved results on standard continuous control benchmarks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Concurrently, Nachum et al. (2018) showed smoothed value functions could be used to train stochastic policies with reduced variance and improved performance.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “Some theoretical work suggests that the ideal expectation is in fact the on-policy expectation (Sutton et al., 2000; Silver et al., 2014; Nachum et al., 2018).”
Learning to Walk via Deep Reinforcement Learning (2018, methodology): “We build on model-free off-policy methods based on value function estimation, since such algorithms generally provide improved sample efficiency compared to on-policy algorithms [2], [3], [4], [5].”
Understanding the impact of entropy on policy optimization (2018, methodology): “The motivation of our work is closely related to Rajeswaran et al. (2017); Henderson et al.”