Batch Stationary Distribution Estimation
By Junfeng Wen, Bo Dai, Lihong Li, Dale Schuurmans. 2020
26 citations recorded by the source corpus.
We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a fixed set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.
Subfields: Management Information Systems, Management Science and Operations Research, Statistics and Probability, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Information Systems
- Mohammad Norouzi
- Sergey Levine
- Alexander Novikov
- Ziyun Wang
- Cosmin Paduraru
- Alekh Agarwal
- Ofir Nachum
In Management Science and Operations Research
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- George Tucker
- Justin Fu
- Mohammad Norouzi
- Mengjiao Yang
- Cosmin Paduraru
In Statistics and Probability
- George Tucker
- Sergey Levine
- Ziyun Wang
- Cosmin Paduraru
- T. Paine
- Mohammad Norouzi
- Michael R. Zhang
- Mengjiao Yang
In Probabilistic & Bayesian Methods
- George Tucker
- Mengjiao Yang
- Ofir Nachum
- Mohammad Norouzi
- Justin Fu
- Aviral Kumar
- Sergey Levine
- Alexander Novikov
Citations in the corpus
- Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Our implementation is based on the same network and hyperparameters for OPE setting as in Wen et al. (2020).”
- Continual Learning In Environments With Polynomial Mixing Times (2021, background): “Even though estimating the steady-state distribution would clearly be quite challenging in complex state and observation spaces, recent approaches have made great strides towards developing practical and scalable approaches leveraging data buffers [24].”
- Provably Good Batch Reinforcement Learning Without Great Exploration (2020, methodology): “[39]) and our algorithmic framework is composable with better estimators of μ(s, a) or other uncertainty meansurements.”
- Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “Our implementation is based on the same network and hyperparameters for OPE setting as in Wen et al. (2020).”
- GFlowNet Foundations (2021, unknown): “Unfortunately, computing stationary distributions, although possible (Nachum et al., 2019; Wen et al., 2020), is not always tractable nor precise enough for purposes of reward regularization.”