Safe Policy Improvement with Soft Baseline Bootstrapping

By Kimia Nadjahi, Romain Laroche, Rémi Tachet des Combes. 2019

37 citations recorded by the source corpus.

Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probability that the trained policy performs better than the behavioural policy, also called baseline in this setting. Previous work shows that the SPI objective improves mean performance as compared to using the basic RL objective, which boils down to solving the MDP with maximum likelihood (Laroche et al. 2019). Here, we build on that work and improve more precisely the SPI with Baseline Bootstrapping algorithm (SPIBB) by allowing the policy search over a wider set of policies. Instead of binarily classifying the state-action pairs into two sets (the uncertain and the safe-to-train-on ones), we adopt a softer strategy that controls the error in the value estimates by constraining the policy change according to the local model uncertainty. The method can take more risks on uncertain actions all the while remaining provably-safe, and is therefore less conservative than the state-of-the-art methods. We propose two algorithms (one optimal and one approximate) to solve this constrained optimization problem and empirically show a significant improvement over existing SPI algorithms both on finite MDPS and on infinite MDPs with a neural network function approximation.

Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Alternatively, we can search over policies in a model of the MDP and bound the error of the estimated model with high probability (Ghavamzadeh et al., 2016; Laroche et al., 2017; Nadjahi et al., 2019).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “SPIBB [33, 41] methods bootstrap using the behavior policy in a Q-learning algorithm for unseen actions.”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “[17], we set the hyper-parameters for the experiments with |D| = 10, 000 (|D| = 3, 000) as follows: Πb -SPIBB withN∧ = 3 (N∧ = 1), Πb -Soft with ε = 0.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “Based on this intuition, a variety of model-based [55, 19, 56] and model-free [22, 33, 11, 34, 25, 37, 44, 14, 28, 23, 3] offline RL algorithms are proposed that achieve promising empirical results.”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “, 2021) estimate of either the Q-value or the bootstrapped Q-value (Kumar et al., 2020; Laroche et al., 2019; Simão et al., 2020; Nadjahi et al., 2019).”