Safe Policy Improvement with an Estimated Baseline Policy

By Thiago D. Simão, Romain Laroche, Rémi Tachet des Combes. 2019

26 citations recorded by the source corpus.

Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Most of these methods require a separately estimated model to the behavior policy, πβ(a|s) [15, 30, 59, 27, 54, 55], and are thus limited by their ability to accurately estimate the unknown behavior policy [42], which might be especially complex in settings where the data is collected from multiple ”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, background): “Although SPIBB-DQN can be extended to settings where the behavioral policy is estimated [Simão et al., 2019], we omit it due to the lack of implementation for a deep setting without access to pseudo-counts of the dataset, which again would require a large number of design and hyper-parameter…”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “, 2021) estimate of either the Q-value or the bootstrapped Q-value (Kumar et al., 2020; Laroche et al., 2019; Simão et al., 2020; Nadjahi et al., 2019).”