Safe Policy Iteration

By Matteo Pirotta, Marcello Restelli, Alessio Pecorino, Daniele Calandriello. 2013

112 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, methodology): “gence constraint, but its purpose is to encourage the policy not to stray from regions where the estimated dynamics model is valid, while we do not attempt to estimate the system dynamics explicitly. Pirotta et al. (2013) also build on and generalize Kakade and Langford’s results, and they derive di”
Constrained Policy Optimization (2017, methodology): “By bounding the expectation to get a second factor of max s D TV ( π (cid:48) || π )[ s ] , we recover (up to assumption-dependent factors) the bounds given by Pirotta et al. (2013) as Corollary 3.6, and by Schulman et al. (2015) as Theorem 1a.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “…application of off-policy value evaluation is to ensure that a new policy to be deployed does not have degenerate performance in policy iteration; example algorithms for this purpose include conservative policy iteration (Kakade & Langford, 2002) and safe policy iteration (Pirotta et al., 2013).”
Safe Policy Improvement by Minimizing Robust Baseline Regret (2016, methodology): “A major limitation of the existing methods for computing safe policies is that they either adopt a newly learned policy with provable improvements or do not make any improvement at all by returning the baseline policy.”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, result): “In fact, many works in the literature have proved similar results under different scenarios [26, 49].”
Doubly Robust Off-policy Evaluation for Reinforcement Learning (2015, background): “In conservative policy iteration (Kakade & Langford, 2002) and safe policy iteration (Pirotta et al., 2013), local adjustments to policies are made to ensure monotonic improvements over iterations.”
Policy Optimization via Importance Sampling (2018, background): “More recently, this idea has been captured (albeit indirectly) by TRPO, which optimizes via (approximate) natural gradient a surrogate objective function, derived from safe RL [17, 34], subject to a constraint on the KullbackLeibler divergence between the behavioral and target policy.”
Non-parametric Policy Search with Limited Information Loss (2017, background): “In conservative policy iteration (Kakade and Langford, 2002) and safe policy iteration (Pirotta et al., 2013), the updated policy is a mixture of the old policy and a greedy policy that maximized a lower bound on the policy improvement.”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “Conventionally, RL algorithms designed to ensure safety [47, 26, 58, 69, 38, 17] model the environment as a Markov decision process (MDP), and rely upon the stationarity assumption made by MDPs [53].”
Adaptive Step-Size for Policy Gradient Methods (2013, background): “The idea is to start from the general lower bound on the performance difference between any pair of policies introduced in [15] and specialize it to the policy gradient framework.”
High Confidence Policy Improvement (2015, background): “To the best of our knowledge, conservative policy iteration (CPI) (Kakade & Langford, 2002) and its derivatives (Pirotta et al., 2013) are the only RL algorithms with monotonically improving behavior.”
A Lyapunov-based Approach to Safe Reinforcement Learning (2018, background): “In particular, besides optimizing performance it is crucial to guarantee the safety of an agent in deployment [5, 32, 33], as well as during training [2].”
Adaptive Batch Size for Safe Policy Gradients (2017, unknown): “Inspired by the conservative policy iteration approach [10], recently, new advances have been done in the field of approximate policy iteration algorithms [11, 12], obtaining methods that can learn faster while still giving statistical guarantees of improvement after each policy update [13, 14].”
Exponentially Weighted Imitation Learning for Batched Historical Data (2018, methodology): “safe policy iteration [Pirotta et al., 2013] has long been an interesting topic in the literature.”
Safe Reinforcement Learning (2015, background): “2 Related Work The only existing safe RL algorithms that we are aware of are conservative policy iteration (CPI) (Kakade, 2003, Kakade and Langford, 2002a) and its derivatives (Pirotta et al., 2013).”