The Ladder: A Reliable Leaderboard for Machine Learning Competitions

By Avrim Blum, Moritz Hardt. 2015

168 citations recorded by the source corpus.

The organizer of a machine learning competition faces the problem of maintaining an accurate leaderboard that faithfully represents the quality of the best submission of each competing team. What makes this estimation problem particularly challenging is its sequential and adaptive nature. As participants are allowed to repeatedly evaluate their submissions on the leaderboard, they may begin to overfit to the holdout data that supports the leaderboard. Few theoretical results give actionable advice on how to design a reliable leaderboard. Existing approaches therefore often resort to poorly understood heuristics such as limiting the bit precision of answers and the rate of resubmission. In this work, we introduce a notion of leaderboard accuracy tailored to the format of a competition. We introduce a natural algorithm called the Ladder and demonstrate that it simultaneously supports strong theoretical guarantees in a fully adaptive model of estimation, withstands practical adversarial attacks, and achieves high utility on real submission files from an actual competition hosted by Kaggle. Notably, we are able to sidestep a powerful recent hardness result for adaptive risk estimation that rules out algorithms such as ours under a seemingly very similar notion of accuracy. On a practical note, we provide a completely parameter-free variant of our algorithm that can be deployed in a real competition with no tuning required whatsoever.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “A number of works have proffered explanations for why adaptive overﬁtting does not occur in machine learning (Blum & Hardt, 2015; Mania et al., 2019; Feldman et al., 2019; Zrnic & Hardt, 2019).”
Do ImageNet Classifiers Generalize to ImageNet? (2019, methodology): “[3] Avrim Blum and Moritz Hardt.”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, methodology): “This mismatch presents a clear danger because the research community could easily be designing models that only work well on the speciﬁc test set but actually fail to generalize to new data [1].”
Calibration for the (Computationally-Identifiable) Masses (2017, methodology): “Additionally, to avoid naively resampling but still guarantee good generalization from a small sample, we interact with the sample through a mechanism which we call a guess-and-check statistical query (similar in spirit to mechanisms proposed in [HR10, BH15, GRU12]).”
Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle (2015, background): “Dwork et al. (2015b) is part of an independent line of work (Hardt & Ullman, 2014; Bassily et al., 2015; Dwork et al., 2015a; Blum & Hardt, 2015) on adaptive data analysis, which also 1For instance, Frank McSherry described in a blog post an example of exploiting differential privacy for measure…”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “Other work has proposed changes to prevent over-fitting via multiple resubmissions (Hardt and Blum, 2015; Hardt, 2017) while some have argued that this issue is overblown, given”
Model Similarity Mitigates Test Set Overuse (2019, background): “Moreover, since our technique is essentially tightening a union bound, this improvement easily extends to other mechanisms that rely on compression-based arguments, for instance, the Ladder Mechanism [2].”
The Benchmark Lottery (2021, methodology): “In such setups, where the discovery of new models is built on top of thousands of queries, direct or indirect, to the test set, the error rate on test data does not necessarily reﬂect the true population error [Arora and Zhang, 2021, Blum and Hardt, 2015, Dwork et al., 2015].”
Neural Network Matrix Factorization (2015, background): “) AutoRec achieves state-of-the-art performance, slightly besting LLORMA on some benchmarks, but a careful comparison would likely require a fresh data set and strict controls on how the numerous parameters for both models are chosen (Blum and Hardt, 2015; Dwork et al., 2015).”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “An active line of research investigates the question of overﬁtting from test set reuse, also known as adaptive overﬁtting [5, 9, 14, 37, 42, 91, 175].”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “, 2019) which—although mitigable (Blum and Hardt, 2015; Anderson-Cook et al., 2019)—is best solved by continually collecting new data (Kiela et al.”
Controlling Bias in Adaptive Data Analysis Using Information Theory (2015, background): “Minimal feedback The recently proposed Ladder Mechanism [3] is an example of the second strategy, of explicitly minimizing the total amount of information in the feedback.”
Automated assessment of psychiatric disorders using speech: A systematic review (2019, background): “multiple-comparison correction across submissions and prioritizing simpler models.(142,143)”
Controlling False Discoveries During Interactive Data Exploration (2016, background): “In [5] Blum and Hardt presented “the Ladder”, an algorithmic test procedure that, given a training dataset, reliably evaluates the quality of different versions a model by adaptively tuning the parameters.”
Natural Analysts in Adaptive Data Analysis (2019, background): “However, in general, this standard choice quickly leads to overﬁtting (see the paper [5] for an example attack).”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “Given that the test data for static benchmarks is often publicly available, reported results may also be the outcome of overﬁtting hyperparameters [4].”