Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models

By B. '. Pires. 2016

23 citations recorded by the source corpus.

In this paper we study a model-based approach to calculating approximately optimal policies in Markovian Decision Processes. In particular, we derive novel bounds on the loss of using a policy derived from a factored linear model, a class of models which generalize numerous previous models out of those that come with strong computational guarantees. For the first time in the literature, we derive performance bounds for model-based techniques where the model inaccuracy is measured in weighted norms. Moreover, our bounds show a decreased sensitivity to the discount factor and, unlike similar bounds derived for other approaches, they are insensitive to measure mismatch. Similarly to previous works, our proofs are also based on contraction arguments, but with the main differences that we use carefully constructed norms building on Banach lattices, and the contraction property is only assumed for operators acting on "compressed" spaces, thus weakening previous assumptions, while strengthening previous results.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “6 This result is known anecdotally (see e.g., Pires & Szepesv´ari, 2016) and we include details in Appendix E for completeness. on these results, and (2) our analyses simplify prior results without trivializing them, making the high-level insights more accessible.”
Iterative Value-Aware Model Learning (2018, methodology): “[2014] who introduce the concept of pseudo-MDP and factored linear action models, which relaxes the constraint that P should be a probability kernel, and Ávila Pires and Szepesvári [2016] who analyzed and provided policy error bounds of such models. We briefly note that work of Ormoneit and Sen [200”
Model-Based Reinforcement Learning with Value-Targeted Regression (2020, background): “That we allow signed distributions only increases generality; this may be important when one is given a model class that can be compactly represented but only when it also includes non-probability kernels (see Pires & Szepesvári 2016 for a discussion of this).”
Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, background): “An alternative is to approximate full dynamic programming updates, to give an expected update, as done by stochastic factorization approaches [Barreto et al., 2011; Kveton and Theocharous, 2012; Barreto et al., 2014; Yao et al., 2014; Barreto et al., 2016; Pires and Szepesvári, 2016], kernel-based R”
Value-Aware Loss Function for Model-based Reinforcement Learning (2017, background): “Ávila Pires and Szepesvári [2016] provide such a policy error bound.”