Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
By Ali Mousavi, Lihong Li, Qiang Liu, Denny Zhou. 2020
33 citations recorded by the source corpus.
Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, \citet{liu18breaking} proposed an approach that avoids the curse of horizon suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data being collected by a known behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a "backward flow" operator and show that the fixed point solution gives the desired importance ratios of stationary distributions between the target and behavior policies. We analyze its asymptotic consistency and finite-sample generalization. Experiments on benchmarks verify the effectiveness of our proposed approach.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Aviral Kumar
- George Tucker
- Justin Fu
- Ofir Nachum
- Shimon Whiteson
- Chelsea Finn
- Shixiang Shane Gu
In Computational Theory and Mathematics
- Sergey Levine
- George Tucker
- Aviral Kumar
- Justin Fu
- Shimon Whiteson
- Shangtong Zhang
- Ofir Nachum
- Bo Liu
In Management Science and Operations Research
- Sergey Levine
- Aviral Kumar
- Justin Fu
- George Tucker
- Shimon Whiteson
- Bo Liu
- Shangtong Zhang
- Chelsea Finn
In Reinforcement Learning
- Sergey Levine
- Aviral Kumar
- George Tucker
- Justin Fu
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Chelsea Finn
Citations in the corpus
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Recent work (Mousavi et al., 2020; Kallus and Uehara, 2019a,b; Tang et al., 2019; Uehara and Jiang, 2019) has refined this approach, in particular removing the need to have access to πβ .”
- GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “We report learning curves of DualDICE, GenDICE, and GradientDICE in Reacher-v2 with the initial learning rate of the Adam optimizer set to 10 − 5 in Figure 6.”
- Representation Balancing Offline Model-based Reinforcement Learning (2021, unknown): “Recently, techniques of computing a stationary DIstribution Correction Estimation (DICE) have made remarkable progress that effectively addresses the curse of horizon (Liu et al., 2018a; Nachum et al., 2019a; Tang et al., 2020; Zhang et al., 2020; Mousavi et al., 2020).”