Optimizing for the Future in Non-Stationary MDPs

By Yash Chandak, Georgios Theocharous, Shiv Shankar, Sridhar Mahadevan, Martha White, Philip S. Thomas. 2020

74 citations recorded by the source corpus.

Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are fixed, that is, the underlying Markov decision process is stationary. However, in many real-world applications, this assumption is violated, and using existing algorithms may result in a performance lag. To proactively search for a good future policy, we present a policy gradient algorithm that maximizes a forecast of future performance. This forecast is obtained by fitting a curve to the counter-factual estimates of policy performance over time, without explicitly modeling the underlying non-stationarity. The resulting algorithm amounts to a non-uniform reweighting of past data, and we observe that minimizing performance over some of the data from past episodes can be beneficial when searching for a policy that maximizes future performance. We show that our algorithm, called Prognosticator, is more robust to non-stationarity than two online adaptation techniques, on three simulated problems motivated by real-world applications.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “Generally speaking, the agent faces a sequence of Markov decision processes {Mz}∞z=1 (Lecarpentier and Rachelson, 2019; Chandak et al., 2020; Khetarpal et al., 2020b) with possibly different transition dynamics, reward functions or even state or action spaces.”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, background): “Within RL, non-stationarity Chandak et al. (2020) study a setting similar to ours, where the reward and transition dynamics change smoothly across episodes, and propose to use curve-ﬁtting to estimate performance on future MDPs and learn a single policy that optimizes for future performance.”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “[13]: While in the past, returns were observed by executing policies {βi}i=1, what if policy π was executed instead? Formally, we use per-decision importance sampling [48] for Hi, to obtain a counterfactual estimate ρ̂(π, i) := ∑∞ t=0 (∏t l=0 π(Ali|S l i) βi(Ai|S i) ) γR i , of π’s performance in th”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “In RL, this setting is analogous to the DP-MDP [66, 10].”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “[2] trains an agent to forecast the future tasks in non-stationary MDPs.”
Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning (2022, background): “Prior work has highlighted the importance of correctly re-using collected experiences in off-policy algorithms when non-stationarities are encountered (Liotet et al., 2021; Chandak et al., 2020; Xie et al., 2021).”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, methodology): “We then consider baselines from the literature, including Pro-OLS and Pro-WLS (Chandak et al. 2020).”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, background): “[7] studies a setting similar to ours, where the reward and transition dynamics change smoothly across episodes, and proposes to use curve-fitting to estimate performance on future MDPs and learns a single policy that optimizes for future performance.”