Stochastic Variance-Reduced Policy Gradient

By Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, Marcello Restelli. 2018

213 citations recorded by the source corpus.

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective func- tion; II) approximations in the full gradient com- putation; and III) a non-stationary sampling pro- cess. The result is SVRPG, a stochastic variance- reduced policy gradient algorithm that leverages on importance weights to preserve the unbiased- ness of the gradient estimate. Under standard as- sumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “A standard approach to dealing with oﬀ-policy data is to correct the policy gradient equation using importance sampling (IS) using the cumulative density ratios, ν 0: t (see, e.g. , Papini et al., 2018, Appendix A; Hanna and Stone, 2018).”
Predictor-Corrector Policy Optimization (2018, methodology): “Finally, we note that the idea of using Φn as control variate [15, 16, 17] is orthogonal to the setups considered above, and it can be naturally combined with PicCoLO.”
Applications of reinforcement learning in energy systems (2021, background): “Thus, in recent years, the RL research community focuses on developing stable RL algorithms with reduced variance [169,170].”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, methodology): “Armed with the advances in non-convex optimization, several research efforts ([57], [58], [59], [60]) propose variants”
Recent advances in reinforcement learning in finance (2021, background): “For more examples of sample complexity analysis for convergence to a stationary point, see, for example, Papini et al. (2018), Xu et al. (2020a, 2020b), Shen et al. (2019), Xiong et al. (2021).”