Protecting against evaluation overfitting in empirical reinforcement learning

By Shimon Whiteson, Brian Tanner, Matthew E. Taylor, Peter Stone. 2011

121 citations recorded by the source corpus.

Empirical evaluations play an important role in machine learning. However, the usefulness of any evaluation depends on the empirical methodology employed. Designing good empirical methodologies is difficult in part because agents can overfit test evaluations and thereby obtain misleadingly high scores. We argue that reinforcement learning is particularly vulnerable to environment overfitting and propose as a remedy generalized methodologies, in which evaluations are based on multiple environments sampled from a distribution. In addition, we consider how to summarize performance when scores from different environments may not have commensurate values. Finally, we present proof-of-concept results demonstrating how these methodologies can validate an intuitively useful range-adaptive tile coding method.

Subfields: Artificial Intelligence, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, background): “by Whiteson et al. (2011). Although we can always report full performance tables, as we did in Appendix D, some more compact summary statistics are also desirable.”
Deep Reinforcement Learning that Matters (2017, background): “Generaliz-able RL evaluation metrics are proposed in (Whiteson et al. 2011).”
What Makes Reading Comprehension Questions Easier? (2018, methodology): “Our heuristics and annotation were motivated by unintended biases (Levesque, 2014) and evaluation overfitting (Whiteson et al., 2011), respectively.”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “This can be attributed to the fact that it is quite common to evaluate an agent on the training environments themselves, which leads to overfitting on these narrowly defined environments (Whiteson et al., 2011), or that algorithms are”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “There has been relatively little work done in RL community on overfitting to the environment [102, 69], but we argue that this is particularly important in multiagent settings where one must react dynamically based on the observed behavior of others.”
A Study on Overfitting in Deep Reinforcement Learning (2018, background): “In Whiteson et al. (2011), a “multiple environments” paradigm is proposed to separate the training and testing stages.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “In single-agent RL, agents can overfit to the environment [296].”
An Introduction to Deep Reinforcement Learning (2018, unknown): “These problems have been commonly used as benchmarks for tabular RL and RL algorithms using linear function approximators (Whiteson et al., 2011).”
RE-EVALUATE: Reproducibility in Evaluating Reinforcement Learning Algorithms (2018, background): “However, with Deep RL, reproducible research is often more difficult [4] due to several factors including intrinsic variance of the algorithm, stochasticity of the environment [5] and dependence on several hyperparameters [6].”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “Overﬁtting and lack of generalization ability have been raised as the most critical problems of deep reinforcement learning (RL) [5, 8, 30, 34, 38, 47, 52].”
Improved empirical methods in reinforcement-learning evaluation (2015, background): “, 2010) and their associated evaluation approach (Whiteson et al., 2011).”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, background): “When solving more practical problems, many issues have been raised, such as ensuring diversity amongst agents [58, 98], avoiding overﬁtting to the policy of the opponents [46, 48, 59, 65, 78].”
Interference and Generalization in Temporal Difference Learning (2020, methodology): “The evaluation methods of new algorithms in the recent union of neural networks and TD, despite an earlier recognition of the problem (Whiteson et al., 2011), often do not include generalization measures, 1We refer here to the recently studied phase transition between generalizing and overfitting,…”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, unknown): “…and tested on the same environment tend to overfit to that environment’s idiosyncracies and are unable to generalize to even small perturbations (Whiteson et al., 2011; Rajeswaran et al., 2017; Zhang et al., 2018c;a; Henderson et al., 2018; Cobbe et al., 2019; Raileanu & Rocktäschel, 2020;…”
Neuroevolutionary reinforcement learning for generalized control of simulated helicopters (2011, background): “To discourage participants from overﬁtting to a single MDP and to encourage them to develop methods capable of robust, on-line learning, many events in the RL Com-petitions are formulated as generalized tasks [91, 92].”
An automated measure of MDP similarity for transfer in reinforcement learning (2014, background): “Unfortunately, this can lead to the problem of empirical overfitting (Whiteson et al. 2011) (i.e., an algorithm can work well given specific MDP parameters values, but not when using other parameter values or different MDPs).”
Autonomous Agents and Multi-agent Systems Manuscript No. Learning Potential Functions and Their Representations for Multi-task Reinforcement Learning (unknown): “This definition matches the generalized environment proposed in [86].”
The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract) (2015, methodology): “Such an evaluation is prone to method overfitting [Whiteson et al., 2011] and discounts the amount of expert effort necessary to transfer the algorithm to new domains.”
ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents (2018, background): “While there has certainly been criticism of the ﬁeld for testing in the training environment [23], there is a legitimate justiﬁcation for doing so: if an agent is only to be deployed in the training environment, it is perfectly reasonable to overﬁt the agent’s policy to that environment.”
The Malmo Platform for Artificial Intelligence Experimentation (2016, background): “[Whiteson et al. , 2011] ﬁrst noted the danger of overﬁt-ting to individual tasks, and proposed evaluation on multiple tasks that are sampled from a distribution to encourage generality.”