Safe Policy Improvement with Baseline Bootstrapping in Factored Environments

By Thiago D. Simão, Matthijs T. J. Spaan. 2019

26 citations recorded by the source corpus.

We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. Factored reinforcement learning, on the other hand, is known to make good use of the data provided. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rémi Tachet des Combes
2. Thiago D. Simão
3. Romain Laroche
4. Kimia Nadjahi
5. Matthijs T. J. Spaan
6. Frans Adriaan Oliehoek
7. Shimon Whiteson
8. Joelle Pineau

In Computational Theory and Mathematics

1. Romain Laroche
2. Rémi Tachet des Combes
3. Kimia Nadjahi
4. Thiago D. Simão
5. Matthijs T. J. Spaan

Citations in the corpus

• Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “Among the existing SPI algorithms, a recent computationally efficient and provablysafe methodology is SPI with Baseline Bootstrapping (SPIBB, Laroche et al., 2019; Simão and Spaan, 2019).”
• Structure Learning for Safe Policy Improvement (2019, methodology): “In previous work [Simão and Spaan, 2019], we show that the Πb-SPIBB framework can be extended to factored environments.”