Bias in Natural Actor-Critic Algorithms

By Philip S. Thomas. 2014

123 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “This parameter corresponds to the discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction parameter in an undiscounted problem; this technique was analyzed theoretically by Marbach & Tsitsiklis (2003); Kakade (2001b); Thomas (2014).”
Memory-based control with recurrent neural networks (2015, methodology): “licitly in the update which is absorbed in the discounted state-visitation distribution in eq. 3. In practice we ignore this term as is often done in policy gradient implementations in practice (e.g. [26]). 3 long episodes or continuing tasks it is possible to use truncated BPTT, although we do not ”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by Thomas (2014).”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by Thomas (2014).”
PGQ: Combining policy gradient and Q-learning (2016, background): “, non-discounted) policy performance, even whenQ uses a discount factor (Thomas, 2014).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A common modification is to drop the γ term in front of the gradient, which approximates an average reward setting (Thomas, 2014).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “, γ = 1 for ρ(s)), which produces a biased estimator for ∇J(θ) and more closely aligns with maximizing average reward (Thomas, 2014).”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “P If the baseline estimates the value function, − Likelihood ratio policy gradient methods use unbiased gradient estimates (except for the technicality detailed by Thomas (2014)), but they often suffer from high variance and are sample-intensive.”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “For example, function approximators (critics) have been adopted to (partially) replace the Monte Carlo estimates of accumulated costs to reduce variance but at the expense of introducing bias in the search direction [2, 11–15].”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “Previous work by Thomas [2014] to remove bias in natural actor-critic algorithms is of interest here, as it suggests weighting actor updates by an accumulating product of discount factors, which can be thought of as an on-policy precursor to emphatic weightings.”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, background): “Nevertheless, in the function approximation case little attention has been paid in the literature to specifing the relative importance of different states (an exception is Thomas 2014), though there are intimations of this in the initiation set of options (Sutton et al. 1999).”
P3O: Policy-on Policy-off Policy Optimization (2019, background): “However, as Thomas (2014) show, being unbiased is not necessarily good and may even hurt performance.”
Soft Actor-Critic Algorithms and Applications (2018, background): “In the context of policy search algorithms, the use of a discount factor is actually a somewhat nuanced choice, and writing down the precise objective that is optimized when using the discount factor is non-trivial (Thomas, 2014).”
Emphatic Temporal-Difference Learning (2015, background): “For example, in episodic problems we often care primarily about the value of the first state, or of earlier states generally (Thomas 2014).”
Optimizing agent behavior over long time scales by transporting value (2018, neutral): “In Supplement Section 6, we show that with γ = 1 the SNR is approximately SNR ≈ ‖Eπ[∆θ]‖ 2 Varπ [∑ t∈P2 rt ] × C(θ) + Varπ[∆θ|no P2] , (6)”
True Online Emphatic TD(λ): Quick Reference and Implementation Guide (2015, methodology): “(Or, as suggested by the work of Thomas (2014), one may want to use I t = γ t , where t here is the time since the beginning of the episode.)”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, unknown): “Here we present the non-discounted setting for brevity and refer to Thomas (2014); Haarnoja et al. (2018) for the full details behind the γ -discounted setting.”
Algorithms for CVaR Optimization in MDPs (2014, background): “[28] P.”
Policy Evaluation Networks (2020, background): “The gradient (1) is typically estimated with samples taken under the undiscounted distribution induced by π θ in the MDP (Thomas, 2014).”