Safe Policy Improvement with Baseline Bootstrapping

By Romain Laroche, Paul Trichelair. 2017

226 citations recorded by the source corpus.

A common goal in Reinforcement Learning is to derive a good strategy given a limited batch of data. In this paper, we adopt the safe policy improvement (SPI) approach: we compute a target policy guaranteed to perform at least as well as a given baseline policy. Our SPI strategy, inspired by the knows-what-it-knows paradigms, consists in bootstrapping the target policy with the baseline policy when it does not know. We develop two computationally efficient bootstrapping algorithms, a value-based and a policy-based, both accompanied with theoretical SPI bounds. Three algorithm variants are proposed. We empirically show the literature algorithms limits on a small stochastic gridworld problem, and then demonstrate that our five algorithms not only improve the worst case scenarios, but also the mean performance.

Subfields: Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Alternatively, we can search over policies in a model of the MDP and bound the error of the estimated model with high probability (Ghavamzadeh et al., 2016; Laroche et al., 2017; Nadjahi et al., 2019).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “[33] suggests similar quadratic dependence on the horizon and an inverse square-root dependence on the counts.”
Off-policy Bandits with Deficient Support (2020, background): “It emphasizes that function approximation equipped with Q-learning can diverge in the off-policy learning setting, making most off-policy RL methods very conservative in extrapolation because of the severe error-propagation issue [7, 16].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “Most notably, prior work in offline RL has been confined almost exclusively to model-free methods [20, 15, 16, 19, 17, 18, 21].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “This evaluation protocol has been used in domains such as continuous control for robotics [Fujimoto et al., 2018, Kumar et al., 2019, Wu et al., 2019], navigation [Laroche et al., 2019], industrial control [Hein et al., 2017], and Atari video games [Agarwal et al., 2019b].”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “Note that in our work, as well as in Laroche et al. (2019), the number of states is set to 50.”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Most closely related to our work is batch-constrained Q-learning (BCQ) [13] and SPIBB [25], which also discuss instability arising from previously unseen actions.”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “[13] end up considering an approximate solution by maximizing the policy in the MLE-MDP while constraining the policy to be approximately improving in the robust MDPs set Ξ.”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Algorithms like BCQ [Fujimoto et al., 2018a] and SPIBB [Laroche et al., 2019] constrain the policy updates to be within the support of the data/behavior.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Some methods have modified the set of valid actions based on counts [Laroche et al., 2019] or the learned behavior policy [Fujimoto et al.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “Following prior works [31], we show that COMBO enjoys a safe policy improvement guarantee.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “These works have often tested their methods on simple MDPs such as grid worlds (Laroche et al., 2017), or fully observed environments were the state of the world is given (Fu et al., 2020; Fujimoto et al., 2018; Wu et al., 2019).”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “In some instances, access to the behavioral policy is assumed [Precup et al., 2001, Thomas and Brunskill, 2016, Petrik et al., 2016, Laroche et al., 2019], but in our experiments, the behavioral policy is treated as unknown.”
Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “…(Garcı́a & Fernández, 2015) is an overloaded term, as it may be considered with respect to parametric uncertainty (Thomas, 2015; Petrik et al., 2016; Laroche et al., 2019), internal uncertainty (Altman, 1999; Carrara et al., 2019), or interruptibility (Orseau & Armstrong, 2016; Guerraoui et al.,…”
Provably Good Batch Reinforcement Learning Without Great Exploration (2020, methodology): “Figure 2: Frequency of converging to π (Success Rate) for different variants of approximated value iteration: AVI, API, BCQL [12] (with different thresholds), SPIBB [21], our algorithms (PPI, PQI).”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, background): “Although prior works have sought to mitigate this challenge (Lange et al., 2012; Laroche et al., 2017; Kalashnikov et al., 2018; Kumar et al., 2019a; 2020), the goal-conditioned setting presents additional issues, since the ofﬂine data was not necessarily collected by a policy attempting to reach…”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “, 2021) estimate of either the Q-value or the bootstrapped Q-value (Kumar et al., 2020; Laroche et al., 2019; Simão et al., 2020; Nadjahi et al., 2019).”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “REM can also be combined with behavior regularization methods such as SPIBB (Laroche et al., 2019) and BCQ (Fujimoto et al., 2019b) to create better offline RL algorithms.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “To solve these issues, offline RL is an excellent choice to deal with practical problems [3, 24, 35, 42, 15, 28, 4, 23, 54, 12], aiming at learning from a fixed dataset without interaction with environments.”