Towards Safe Policy Improvement for Non-Stationary MDPs

By Yash Chandak, Emma Jordan, Georgios Theocharous, Martha White, Philip S. Thomas. 2020

46 citations recorded by the source corpus.

Many real-world sequential decision-making problems involve critical systems with financial risks and human-life risks. While several works in the past have proposed methods that are safe for deployment, they assume that the underlying problem is stationary. However, many real-world problems of interest exhibit non-stationarity, and when stakes are high, the cost associated with a false stationarity assumption may be unacceptable. We take the first steps towards ensuring safety, with high confidence, for smoothly-varying non-stationary decision problems. Our proposed method extends a type of safe algorithm, called a Seldonian algorithm, through a synthesis of model-free reinforcement learning with time-series analysis. Safety is ensured using sequential hypothesis testing of a policy's forecasted performance, and confidence intervals are obtained using wild bootstrap.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Universal Off-Policy Evaluation (2021, methodology): “[16], we assume that the distribution of returns for any π changes smoothly over the past episodes 1 to L, and the ` episodes in the future.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “…(e.g., legged robots or other contact dynamics with the environment (150)), time-varying dynamics (e.g., operation in changing environments (151, 152)), time delays (e.g., in actuation, sensing, or observing the reward (6)), partial diﬀerential dependencies in the dynamics (e.g., in continuum…”