Batch Policy Learning under Constraints

By Hoang Minh Le, Cameron Voloshin, Yisong Yue. 2019

401 citations recorded by the source corpus.

When learning policies for real-world domains, two important questions arise: (i) how to efficiently use pre-collected off-policy, non-optimal behavior data; and (ii) how to mediate among different competing objectives and constraints. We thus study the problem of batch policy learning under multiple constraints, and offer a systematic solution. We first propose a flexible meta-algorithm that admits any batch reinforcement learning and online learning procedure as subroutines. We then present a specific algorithmic instantiation and provide performance guarantees for the main objective and all constraints. To certify constraint satisfaction, we propose a new and simple method for off-policy policy evaluation (OPE) and derive PAC-style bounds. Our algorithm achieves strong empirical results in different domains, including in a challenging problem of simulated car driving subject to multiple constraints such as lane keeping and smooth driving. We also show experimentally that our OPE method outperforms other popular OPE techniques on a standalone basis, especially in a high-dimensional setting.

Subfields: Artificial Intelligence, Automotive Engineering, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “An alternative but closely related approach is to fit Qπe directly from data using standard approximate dynamic programming (ADP) techniques, e.g., the policy evaluation analog of Fitted Q-Iteration (Ernst et al., 2005; Le et al., 2019).”
Learning When-to-Treat Policies (2019, methodology): “Another variant of the fitted-Q method evaluates a given policy π instead of learning the best policy (see, e.g., Le, Voloshin, and Yue 2019).”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Finally, all evaluations in Voloshin et al. (2019) are performed using the MSE metric, and they do not provide standardized datasets.”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “Although this is a conceptually very simple approach, the earliest explicit reference to using it for OPE seems to be from Le et al. (2019), who refer to it as Fitted Q Evaluation (FQE).”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, methodology): “, 2007) or estimate the q-function directly using fitted LSTDQ or more general q-iteration (Antos et al., 2008; Lagoudakis and Parr, 2004; Le et al., 2019) (we further review estimation of q-functions in Section 4.”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations [31, 57, 3, 1], and the increasing importance of off-policy learning and counterfactual reasoning [12, 52, 37, 30, 34, 41].”
Near-Optimal Offline Reinforcement Learning via Double Variance Reduction (2021, background): “Note that this is weaker than the standard “concentrability” assumption (Munos, 2003; Le et al., 2019; Chen & Jiang, 2019): Concentrability defines μ := sup⇡2⇧ ||d(st, at)/d(st, at)||1 < 1 (cf.”
Preventing undesirable behavior of intelligent machines (2019, background): “[55] propose reinforcement learning algorithms that allow the user to specify a “safe” set of policies, and guarantee that the algorithms will always converge to policies contained in this safe set.”
Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions (2022, methodology): “Not only do the limitations of IS and FQE on return estimation carry over to this more challenging task, but MIS also loses its major advantage over FQE: despite the somewhat misleading impression left by many prior works, that MIS can handle function estimation the same way as return estimation, 1 ”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, methodology): “Fitted Q-Evaluation (FQE) (Ernst et al., 2005; Le et al., 2019) is an iterative method with a similar update:”
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “One of the most common ways to operationalize this is using fitted q-iteration (Antos et al., 2008; Le et al., 2019; expressed here using an estimated evaluation policy, π̂e): • Set q̂H+1 ≡ 0.”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “Figures 4 and B.4 compare OPE estimates from two Fitted-Q Evaluation (FQE) baselines (Le et al., 2019; Kostrikov and Nachum, 2020; Paine et al., 2020), our feedforward models, and the autoregressive approach.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “Online policy evaluation refers to directly run the trained policy in the original simulator or the production environment, while ofﬂine evaluation uses off-policy evaluation (OPE) methods [Voloshin et al. , 2019] or builds an evaluator through Bellman backup [Paine et al. , 2020].”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, background): “Fitted Q-Evaluation (Le et al., 2019) can be thought of as dynamic programming on an observational dataset to compute the value of a given evaluation policy.”
Evolution of Non-Terrestrial Networks From 5G to 6G: A Survey (2021, background): “Towards this goal, the available literature on safe RL [280], batch RL [281], [282] and”
Provable Representation Learning for Imitation Learning via Bi-level Optimization (2020, background): “Le et al. [2019] used a similar notion that they call inherent Bellman evaluation error .”
A Distributional View on Multi-Objective Policy Optimization (2020, background): “, 2018); this has also been studied in the off-policy batch RL setting (Le et al., 2019).”