Robust Policy Optimization with Baseline Guarantees

By Yinlam Chow, Marek Petrik, Mohammad Ghavamzadeh. 2015

11 citations recorded by the source corpus.

Our goal is to compute a policy that guarantees improved return over a baseline policy even when the available MDP model is inaccurate. The inaccurate model may be constructed, for example, by system identification techniques when the true model is inaccessible. When the modeling error is large, the standard solution to the constructed model has no performance guarantees with respect to the true model. In this paper we develop algorithms that provide such performance guarantees and show a trade-off between their complexity and conservatism. Our novel model-based safe policy search algorithms leverage recent advances in robust optimization techniques. Furthermore we illustrate the effectiveness of these algorithms using a numerical example.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation (2016, positive): “High confidence off-policy model-based methods require large amounts of data and are limited to discrete settings [4].”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, background): “Another class of OPE methods learn a separate model to simulate trajectories in order to estimate policy values (e.g. Chow et al. (2015)), but may suffer from unacceptably high bias in real-world, noisy settings.”
High Confidence Off-Policy Evaluation with Models (2016, background): “used ideas from robust optimization to bound model bias caused by error in a discrete MDP’s transition function and showed that tight lower bounds can be proven on the performance of a policy [4].”