Expected Policy Gradients

By Kamil Andrzej Ciosek, Shimon Whiteson. 2017

60 citations recorded by the source corpus.

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates across the action when estimating the gradient, instead of relying only on the action in the sampled trajectory. We establish a new general policy gradient theorem, of which the stochastic and deterministic policy gradient theorems are special cases. We also prove that EPG reduces the variance of the gradient estimates without requiring deterministic policies and, for the Gaussian case, with no computational overhead. Finally, we show that it is optimal in a certain sense to explore with a Gaussian policy such that the covariance is proportional to the exponential of the scaled Hessian of the critic with respect to the actions. We present empirical results confirming that this new form of exploration substantially outperforms DPG with the Ornstein-Uhlenbeck heuristic in four challenging MuJoCo domains.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Generalized Off-Policy Actor-Critic (2019, methodology): “There have been numerous applications of OPPG including DDPG, ACER, IPG, EPG and IMPALA.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “This difference is so extreme that in several instances the PPO baseline converges an order of magnitude slower than the off-policy algorithms and we thus indicate the asymptotic performance of each algorithm of PPO and DDPG (which also improved signiﬁcantly later during training in some instances) ”
Bootstrapped Policy Gradient for Difficulty Adaptation in Intelligent Tutoring Systems (2019, background): “The trade-off between exploration and exploitation is a tricky issue [4].”
Flexible Option Learning (2021, background): “As such, they are reminescent of generalizations of standard RL algorithms, such Expected SARSA [van Seijen et al., 2009] and Expected Policy Gradients [Ciosek and Whiteson, 2018].”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, methodology): “The main contributions of this work are: 1) an exact reduction of entropy regularised RL to probabilistic inference using value function estimators; 2) the introduction of a theoretically justiﬁed general framework for developing inference-style algorithms for RL that incorporate the uncertainty in ”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “The choice of exploration strategy plays a substantial role in the performance of deep deterministic policy gradient algorithms (Ciosek & Whiteson, 2018).”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, background): “…Gradient (DPG, Silver et al. 2014), Deep Deterministic Policy Gradient (DDPG, Lillicrap et al. 2015), Actor Critic with Experience Replay (ACER, Wang et al. 2016), off-policy Expected Policy Gradient (EPG, Ciosek & Whiteson 2017), TD3 (Fujimoto et al., 2018), and IMPALA (Espeholt et al., 2018).”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “an advantage actor-critic algorithm differing from A2C in the (state-action) representation of the critics [56, 96] and summing over actions similar to the all-action algorithms [86, 71, 19, 5].”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, background): “These include generalized advantage estimation [7], constrained updates [8], updating the target policy via the expectation of its actions [9, 10], and updating the value function via the posterior mean of an estimated uncertain value distribution [11].”
Smoothed Action Value Functions for Learning Gaussian Policies (2018, methodology): “…reward objective (Equation (4)) for a Gaussian policy π ≡ ( µ, Σ) as, The insight that differentiates this approach from prior work (Heess et al., 2015; Ciosek & Whiteson, 2018) is that instead of learning a function approximator for Q π then drawing samples to approximate the expectation…”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, background): “2017), Off-policy Expected Policy Gradients (Ciosek and Whiteson, 2017), and IMPALA (Espeholt et al.”
Expected Policy Gradients for Reinforcement Learning (2018, methodology): “This article, which extends our previous work (Ciosek and Whiteson, 2018), proposes a new approach called expected policy gradients (EPG) that uniﬁes policy gradients in a way that yields both theoretical and practical insights.”