Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes

By Nathan Kallus, Masatoshi Uehara. 2019

52 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “, 2020; Kallus and Uehara, 2019a,b; Tang et al., 2019; Uehara and Jiang, 2019) has refined this approach, in particular removing the need to have access to πβ . Once ρ∗ is obtained, Liu et al. (2019) use this estimator to compute the off-policy policy gradient.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “Applying the Lagrangian to (82) will result in objectives similar to those that have appeared in Kallus and Uehara (2019b); Tang et al. (2019).”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “Here we show that in the tabular setting, which is a special case of the linear setting, MWL and MQL can achieve the semiparametric lower bound of OPE (Kallus and Uehara, 2019a), because they coincide 10See Bartlett and Mendelson (2003) for the definition.”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “Independent of this work, Kallus & Uehara (2019b) extend the technique to the infinite horizon setting, whose estimator can be viewed as a variant of ours.”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, positive): “Recently, this approach to efficient OPE has been extended to Markov decision process [10] and infinite-horizon problems [11]; in this paper we focus on the more general sequential decision process.”
Statistically Efficient Off-Policy Policy Gradients (2020, background): “Following similar arguments as in Kallus & Uehara (2019a), which study infinite-horizon OPE, one can show that this extension of EOPPG maintains its efficiency and 3-way robustness guarantees as long as our data satisfies appropriate mixing conditions (which ensures it appropriately approximates…”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, background): “This is related to but different from the usual notion of double robustness in RL [15, 3, 16, 17]: classical doubly robust methods are typically “meta”-estimators and require a value-function whose estimation procedure is unspecified, and the double robustness refers to the fact that the estimation ”
Near-Optimal Offline Reinforcement Learning via Double Variance Reduction (2021, background): “One major line of work is concerned with the off-policy evaluation (OPE) problem [Li et al., 2015, Jiang and Li, 2016, Liu et al., 2018, Kallus and Uehara, 2019a,b, Uehara and Jiang, 2019, Xie et al., 2019, Yin and Wang, 2020, Duan and Wang, 2020].”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “Independent of this work, Kallus & Uehara (2019b) extend the technique to the inﬁnite horizon setting, whose estimator can be viewed as a variant of ours.”
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “(20) Here, the terms from (18) to (19) are the same as the decomposition in Kallus and Uehara (2019a).”
Off-Policy Evaluation via the Regularized Lagrangian (2020, background): “Recently, a series of works were proposed to address these challenges [14, 17, 26].”