Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines

By Philip S. Thomas, Emma Brunskill. 2017

60 citations recorded by the source corpus.

We show how an action-dependent baseline can be used by the policy gradient theorem using function approximation, originally presented with action-independent baselines by (Sutton et al. 2000).

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Several recent methods (Gu et al., 2017b; Thomas & Brunskill, 2017; Grathwohl et al., 2018; Liu et al., 2018; Wu et al., 2018) have extended the approach to state-action-dependent baselines (i.e., φ ( s, a ) is a function of the state and the action).”
Action-depedent Control Variates for Policy Optimization via Stein's Identity (2017, methodology): “Besides Q-prop (Gu et al., 2016b) which we draw close connection to, the work of Thomas & Brunskill (2017) also suggests a way to incorporate action-dependent baselines, but is restricted to the case of compatible function approximation.”
Stochastic Variance-Reduced Policy Gradient (2018, background): “For this reason, literature has focused on the introduction of baselines (i.e., functions b : S × A → R ) aiming to reduce the variance (e.g., Williams, 1992; Peters & Schaal, 2008a; Thomas & Brunskill, 2017; Wu et al., 2018), see Appendix A for a formal deﬁnition of b .”