Stein Variational Policy Gradient

By Yang Liu, Prajit Ramachandran, Qiang Liu, Jian Peng. 2017

153 citations recorded by the source corpus.

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.

Subfields: Artificial Intelligence, Demography, Management Science and Operations Research, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Yoshua Bengio
4. Chelsea Finn
5. Qiang Liu
6. Igor Mordatch
7. Jeff Clune
8. Bo Dai

In Demography

1. Yilun Du
2. Igor Mordatch
3. Dilin Wang
4. Qiang Liu

In Management Science and Operations Research

1. Sergey Levine
2. Bo Dai
3. Lihong Li
4. Qiang Liu
5. Pieter Abbeel
6. Igor Mordatch
7. Josh Tenenbaum
8. Niao He

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Lihong Li
4. Pieter Abbeel
5. Igor Mordatch
6. Bo Dai
7. Chelsea Finn
8. Shimon Whiteson

In Probabilistic & Bayesian Methods

1. Sergey Levine
2. Pieter Abbeel
3. Yilun Du
4. Stefano Ermon
5. Qiang Liu
6. Igor Mordatch
7. Josh Tenenbaum
8. Bo Dai

Citations in the corpus

• Reinforcement Learning with Deep Energy-Based Policies (2017, background): “A recent work (Liu et al., 2017) also considers an entropy regularized objective, though the entropy is on policy parameters, not on sampled actions.”
• Bayesian Model-Agnostic Meta-Learning (2018, methodology): “Liu et al. (2017) showed that the maximum entropy policy optimization can be recast to Bayesian inference.”
• Smoothed Dual Embedding Control (2017, background): “nt descent sampler, but limited theory is provided. Another related algorithm is developed by Asadi & Littman (2017) for the tabular case, which resembles SARSA with a particular policy; also see Liu et al. (2017) for a Bayesian variant. Observing the duality connection between soft Q-learning a”
• Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “The computation of the entropy term H [ q θ ] can resolved by approaches (Liu et al., 2017) propose an optimization procedure where this term is minimized by construction, but rely on a kernel function κ ( x , x (cid:48) ) , which requires domain-specific design.”
• Distributionally Adversarial Attack (2018, background): “First note that under the H -Wasserstein distance metric defined by (Liu et al. 2017), we have where K ( · , · ) is a kernel function such as the RBF kernel.”
• Improved Contrastive Divergence Training of Energy Based Models (2020, methodology): “Other related entropy estimators include those based on Stein’s identity (Liu et al., 2017) and MINE (Belghazi et al.”
• Implicit Generation and Modeling with Energy Based Models (2019, methodology): “The computation of the entropy term H [ q θ ] can resolved by approaches [Liu et al., 2017] propose an optimization procedure where this term is minimized by construction, but rely on a kernel function κ ( x , x (cid:48) ) , which requires domain-specific design.”
• Active Domain Randomization (2019, methodology): “We use a temperature α = 10 with an RBF-Kernel as was done in (Liu et al., 2017).”
• A Bayesian Approach to Generative Adversarial Imitation Learning (2018, methodology): “We think our algorithm can be simply extended to the multi-policy imitation learning, and the sample efficiency of our algorithm may be enhanced by utilizing the posterior of the policy parameter as shown in Stein variational policy gradient (SVPG) [Liu et al., 2017].”
• Stein Variational Gradient Descent as Moment Matching (2018, methodology): “However, the results in Liu et al. (2017); Lu et al. (2018) are asymptotic in nature and hold only when the number of particles is very large.”
• Implicit Generation and Generalization with Energy Based Models (2018, methodology): “Methods such as [Liu et al., 2017] may be helpful in simultaneously minimizing both terms.”
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents (2017, background): “[31], which optimizes for a distribution of M diverse, high-performing policies.”
• Understanding MCMC Dynamics as Flows on the Wasserstein Space (2019, unknown): “, 2019) and application (Liu et al., 2017; Pu et al., 2017; Zhuo et al., 2018; Yoon et al., 2018).”
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, unknown): “…DQN towards better value estimation and [30] combines the previous efforts to demonstrate a strong baseline; Along the policy-based track, [62, 63, 46, 64] develop deep policy gradient methods on top of [67]; And along the AC track, [47, 76, 81, 17, 24] empower both the actor and critic using…”
• Advances in Variational Inference (2017, background): “the RL setting as the Stein policy gradient [110].”