Action-dependent Control Variates for Policy Optimization via Stein Identity

By Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, Qiang Liu. 2018

52 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Several recent methods (Gu et al., 2017a; Thomas & Brunskill, 2017; Grathwohl et al., 2018; Liu et al., 2018; Wu et al., 2018) have extended the approach to state-action-dependent baselines (i.”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “(2019) noticed that the previous variance reduction methods in PG overlooked the correlation across the times steps and ignored the randomness in the future steps (e.g., Grathwohl et al., 2018; Gu et al., 2017; Liu et al., 2018; Wu et al., 2018).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, methodology): “Another line of research uses the control variate (CV) method from statistics, designed for reducing variance in Monte Carlo methods without introducing bias [2, 17–24].”
Reward-estimation variance elimination in sequential decision processes (2018, background): “More recently state-action-dependent baselines have been investigated [2, 4, 8, 20].”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “Our off-policy setting is related to, but different from, off-policy value-function learning [30, 29, 37, 12, 25, 21].”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “Action-dependant baselines had been proposed [57, 58], however, a recent study by Tucker et al.”
Bootstrapped Policy Gradient for Difficulty Adaptation in Intelligent Tutoring Systems (2019, methodology): “Unlike previous works that endeavor to maintain unbiased gradient estimation [8, 11, 21, 29, 31, 32], this paper proved it is legitimate to use biased gradient, as long as the proposed sufficient condition C.”
Neural Approaches to Conversational AI (2018, unknown): “Interested readers can refer to a few related works and the references therein for further details (Kakade, 2001; Peters et al., 2005; Schulman et al., 2015a,b; Mnih et al., 2016; Gu et al., 2017; Dai et al., 2018a; Liu et al., 2018b).”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “an advantage actor-critic algorithm differing from A2C in the (state-action) representation of the critics [56, 96] and summing over actions similar to the all-action algorithms [86, 71, 19, 5].”
Adversarial Contrastive Estimation (2018, methodology): “More advanced gradient estimators exist that also reduce variance (Grathwohl et al., 2017; Tucker et al., 2017; Liu et al., 2018), but for simplicity we use the self-critical baseline method (Rennie et al., 2016), where the baseline is b ( x ) = l ω ( y + , y (cid:63) , x ) , or b ( x ) = − ˜ s ω (…”