Off-policy Model-based Learning under Unknown Factored Dynamics

By Assaf Hallak, François Schnitzler, Timothy A. Mann, Shie Mannor. 2015

28 citations recorded by the source corpus.

Off-policy learning in dynamic decision problems is essential for providing strong evidence that a new policy is better than the one in use. But how can we prove superiority without testing the new policy? To answer this question, we introduce the G-SCOPE algorithm that evaluates a new policy based on data generated by the existing policy. Our algorithm is both computationally and sample efficient because it greedily learns to exploit factored structure in the dynamics of the environment. We present a finite sample analysis of our approach and show through experiments that the algorithm scales well on high-dimensional problems with few samples.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, methodology): “These methods use algorithms based on importance sampling (Precup, 2000; Thomas et al., 2015; Guo et al., 2017), action-value function approximation (model based) (Hallak et al., 2015) or doubly robust combination of both approaches (Jiang & Li, 2015).”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, methodology): “For instance, the GSCOPE algorithm performs effective policy evaluation when the environment has a factored structure [6].”
Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation (2017, methodology): “In contrast to model based approaches to OPE [5], importance sampling provides an unbiased estimate of the performance of the evaluation policy.”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, methodology): “…MDP (for example, by factorizing the state transition model) has also been investigated extensively in the literature (Boutilier et al., 1999; Degris et al., 2006; Strehl et al., 2007; Ross & Pineau, 2008; Bellemare et al., 2013; Wu et al., 2014; Osband & Van Roy, 2014; Hallak et al., 2015).”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Due to performing model estimation in addition to value function estimation, these algorithms tend to be computationally more complex (Dud´ık et al. 2011, Paduraru 2013, Hallak et al. 2015b, Li et al. 2015, Jiang & Li 2015, Thomas & Brunskill 2016).”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “Previous instances of the use of function approximation for model-based OPE (Hallak et al., 2015) impose strong assumptions on the probabilistic dynamics models, such as factorability of the MDP.”
Safe Policy Improvement with Baseline Bootstrapping in Factored Environments (2019, methodology): “Methods have been proposed to improve the conﬁdence in off-policy evaluation algorithms using both ﬂat and factored representations of the problem (Thomas, Theocharous, and Ghavamzadeh 2015a; Hallak et al. 2015).”