Policy gradient in Lipschitz Markov Decision Processes

By Matteo Pirotta, Marcello Restelli, Luca Bascetta. 2015

101 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Tengyu Ma
3. Pieter Abbeel
4. Ofir Nachum
5. Trevor Darrell
6. Yuanzhi Li
7. Chelsea Finn
8. Yuandong Tian

In Control and Systems Engineering

1. Tengyu Ma
2. Emmanuel Rachelson
3. Erwan Lecarpentier
4. Huazhe Xu
5. Sergey Levine
6. Ofir Nachum
7. Yuandong Tian
8. Neev Parikh

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Ofir Nachum
4. Tengyu Ma
5. Marc G. Bellemare
6. Carles Gelada
7. Huazhe Xu
8. Saurabh Kumar

In Statistical Learning Theory

1. Yuanzhi Li
2. Tengyu Ma
3. Emmanuel Rachelson
4. Erwan Lecarpentier
5. Trevor Darrell
6. Marcello Restelli
7. Marc G. Bellemare
8. Yuandong Tian

Citations in the corpus

• DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Pirotta et al. (2015) suggest that the smaller the Lipschitz constant of an MDP, the faster it is to converge to a near-optimal policy.”
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “(See (Jaksch et al., 2010; Bartlett & Tewari, 2009; Fruit et al., 2018; Lakshmanan et al., 2015; Pirotta et al., 2015; 2013) and references therein for the OFU principle in finite-state MDPs.) The main novelty here is to optimize the lower bound directly, without explicitly building any confidence…”
• Stochastic Variance-Reduced Policy Gradient (2018, methodology): “To deal with non-concavity, we require J ( θ ) to be L -smooth, which is a reasonable assumption for common policy classes such as Gaussian 3 and softmax (e.g., Furmston & Barber, 2012; Pirotta et al., 2015).”
• Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “Smoothness is another desirable property and its benefits in RL are well known (Pazis & Parr, 2013; Pirotta et al., 2015; Asadi et al., 2018).”
• Non-Stationary Markov Decision Processes a Worst-Case Approach using Model-Based Reinforcement Learning (2019, methodology): “Utilizing Lipschitz continuity to infer bounds on a function is common in the RL, bandit and optimization communities [Kleinberg et al., 2008, Rachelson and Lagoudakis, 2010, Pirotta et al., 2015, Pazis and Parr, 2013, Munos, 2014].”
• Lipschitz Continuity in Model-based Reinforcement Learning (2018, unknown): “…& Szepesv ´ ari, 1996; M ¨ uller, 1996; Ferns et al., 2004; Hinderer, 2005; Rachelson & Lagoudakis, 2010; Szepesv´ari, 2010; Pazis & Parr, 2013; Pirotta et al., 2015; Pires & Szepesv ´ ari, 2016; Berkenkamp et al., 2017; Bellemare et al., 2017) and bandits (Kleinberg et al., 2008; Bubeck et…”
• Adaptive Batch Size for Safe Policy Gradients (2017, background): “Up to now, works on safe policy gradients [15, 16] have focused mainly on the choice of the step size, a parameter that significantly affects the speed and stability of gradient methods.”