Hyperparameter Selection for Offline Reinforcement Learning

By T. Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyun Wang, Nando de Freitas. 2020

163 citations recorded by the source corpus.

Offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. However, existing hyperparameter selection methods for offline RL break the offline assumption by evaluating policies corresponding to each hyperparameter setting in the environment. This online execution is often infeasible and hence undermines the main aim of offline RL. Therefore, in this work, we focus on \textit{offline hyperparameter selection}, i.e. methods for choosing the best policy from a set of many policies trained using different hyperparameters, given only logged data. Through large-scale empirical evaluation we show that: 1) offline RL algorithms are not robust to hyperparameter choices, 2) factors such as the offline RL algorithm and method for estimating Q values can have a big impact on hyperparameter selection, and 3) when we control those factors carefully, we can reliably rank policies across hyperparameter choices, and therefore choose policies which are close to the best policy in the set. Overall, our results present an optimistic view that offline hyperparameter selection is within reach, even in challenging tasks with pixel observations, high dimensional action spaces, and long horizon.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “Prior studies on machine-generated datasets have shown that offline policy learning can be extremely sensitive to hyperparameter choices [19, 28].”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Following prior work on continuous state and action problems, we always evaluate by simple ﬁtted Q evaluation [Fujimoto et al., 2018a, Kumar et al., 2019, Siegel et al., 2020, Wang et al., 2020b, Paine et al., 2020, Wang et al., 2021].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[45] T.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “We tried two different implementations, one from Kostrikov & Nachum (2020)3 and another from Paine et al. (2020) labeled FQE-L2 and FQE-D respectively to reflect different choices in loss function and parameterization.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Note that this can—and often does—make use of an environment for evaluation purposes (Paine et al., 2020), although as noted above this can be replaced with an arbitrary evaluation mechanism.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “Hyperparameter tuning [65] and OPE are two problems that still lack a satisfying solution.”
Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation (2020, methodology): “We present the computed intervals of FQE and MB in Figure 3.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, result): “Offline policy evaluation (OPE) extends off-policy policy evaluation (OPPE) to offline setting [13, 14] and only uses the dataset to evaluate a policy, thus it can be applied as a tool for offline policy validation.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “Hyperparameter optimization is especially important in the offline RL setting, as evaluating different hyperparameter configurations requires interacting with the environment [33].”
Model-Based Offline Planning (2020, background): “[19] T.”