Toward Minimax Off-policy Value Estimation

By Lihong Li, R´emi Munos, Csaba Szepesvari. 2015

145 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, background): “Note that some theoretical ﬁndings (Li et al., 2015) support that using a regression estimate may indeed improve importance sampling.”
Safe and Efficient Off-Policy Reinforcement Learning (2016, positive): “This may actually even lead to a better estimate, as analyzed by Li et al. (2015).”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “In fact, in the worst case, the amount of data required by off-policy value evaluation is exponential in the horizon (Li et al., 2015).”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “Importance reweighting, or inverse propensity scoring, has been used for off-policy RL [12,18,28,29,32,39].”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “, 2017) and more general finite horizon RL settings (Fonteneau et al., 2013; Li et al., 2015; Jiang & Li, 2016; Thomas & Brunskill, 2016; Liu et al., 2018b; Farajtabar et al., 2018; Hanna et al., 2019; Xie et al., 2019).”
Behavior Regularized Offline Reinforcement Learning (2019, background): “This setting is of particular interest for applications in which deploying a policy is costly or there is a safety concern with updating the policy online (Li et al., 2015).”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, background): “…policy, for a sequential decision-making problem using data generated by the log of another policy, known as the behavior policy (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016; Xie et al., 2019).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “The problem of off-policy policy evaluation has been heavily studied in contextual bandits [12, 49, 52] and in the more general RL setting [14, 21, 26, 29, 34, 36, 37, 50, 51].”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, background): “[8] L.”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “off-policy value estimation as called by some authors [20]).”
Optimal and Adaptive Off-policy Evaluation in Contextual Bandits (2016, background): “Under such consistency assumptions, it has been shown that DM and/or DR are optimal (Imbens et al., 2007; Li et al., 2015; Rothe, 2016),1 whereas standard IPS is not (Hahn, 1998; Li et al.”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, background): “OPE is a key problem in reinforcement learning (RL) (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016) and it finds applications as varied as healthcare (Murphy, 2003) and education (Mandel et al.”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, methodology): “Note that our estimator of r πt ( s t ) is the standard IS estimators we use in bandits [Li et al., 2015], which are shown to be optimal (up to a universal constant) when A is large [Wang et al., 2017].”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “Long horizon? It is well-known that IPS methods are sensitive to trajectory length [32].”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, methodology): “We can rewrite REG as an importance sampling method with a count-based estimate of the probability of a trajectory in the denominator: The denominator in (13) can be re-written as a telescoping product to obtain an estimator that is similar to RIS( L − 1 ): This expression differs from RIS( L − 1 ) ”
Counterfactual Risk Minimization (2015, background): “have been developed for off-policy evaluation. The inverse propensity scoring estimator can be optimal when we have a good model of the historical algorithm (Strehl et al., 2010), (Li et al., 2014), (Li et al., 2015), and doubly robust estimators are even more efﬁcient when we additionally have a go”