Adaptive Step-Size for Policy Gradient Methods

By Matteo Pirotta, Marcello Restelli, Luca Bascetta. 2013

79 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “(See (Jaksch et al., 2010; Bartlett & Tewari, 2009; Fruit et al., 2018; Lakshmanan et al., 2015; Pirotta et al., 2015; 2013) and references therein for the OFU principle in ﬁnite-state MDPs.) The main novelty here is to optimize the lower bound directly, without explicitly building any conﬁdence…”
Multi-agent query reformulation: Challenges and the role of diversity (2019, methodology): “Reinforcement learning algorithms that use non-linear function approximators, such as neural networks, are known to be unstable (Tsitsiklis & Van Roy, 1996; Fairbank & Alonso, 2011; Pirotta et al., 2013; Mnih et al., 2015).”
Expected Policy Gradients (2017, methodology): “riance of the gradient would otherwise explode, using, e.g., Adam (Kingma and Ba, 2014), natural policy gradients (Kakade, 2002; Amari, 1998; Peters and Schaal, 2008a), the adaptive step size method (Pirotta, Restelli, and Bascetta, 2013) or Newton’s method (Furmston and Barber, 2012; Parisi, Pirott”
A Lyapunov-based Approach to Safe Reinforcement Learning (2018, background): “In particular, besides optimizing performance it is crucial to guarantee the safety of an agent in deployment [5, 32, 33], as well as during training [2].”
Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation (2018, methodology): “Reinforcement learning algorithms that use non-linear function approximators, such as neural networks, are known to be unstable (Tsitsiklis & Van Roy, 1996; Fairbank & Alonso, 2011; Pirotta et al., 2013; Mnih et al., 2015).”
Expected Policy Gradients for Reinforcement Learning (2018, methodology): “…Amari, 1998; Peters and Schaal, 2008a), trust region policy optimisation (Schulman et al., 2015), proximal policy optimisation (Schulman et al., 2017), the adaptive step size method (Pirotta et al., 2013) or Newton’s method (Furmston and Barber, 2012; Furmston et al., 2016; Parisi et al., 2016).”
Policy gradient in Lipschitz Markov Decision Processes (2015, methodology): “Moreover, Pirotta et al. (2013) have empirically shown that AGSS is more robust than standard stepsize selectionmechanisms (e.g., constant or time varying) to changes of theMDP parameters.”
Adaptive Batch Size for Safe Policy Gradients (2017, methodology): “the adaptive scalar step size proposed in [15] for the current setting: Corollary 3.”
Stochastic Variance-Reduced Policy Gradient (2018, background): “There is a constant V ∞ such that, for any policy π θ : As mentioned, (Pirotta et al., 2013) provides a bound on the variance of the REINFORCE estimator, adapted from (Zhao et al., 2011), which does not depend on θ : The same authors provide a similar bound for G(PO)MDP. double the…”