The reusable holdout: Preserving validity in adaptive data analysis

By Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth. 2015

367 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, background): “Beyond privacy, training using DP-SGD offers advantages such as strong generalization and the promise of reusable holdouts (Dwork et al. 2015).”
Differential Privacy for Growing Databases (2018, neutral): “A recent line of work [DFH + 15, CLN + 16, BNS + 16] showed deep connections between diﬀerential privacy and adaptive data analysis of a static database.”
Advanced Probabilistic Couplings for Differential Privacy (2016, neutral): “Indeed, differential privacy behaves well under adaptivity, a highly useful property enabling applications to adaptive data analysis and statistics [23].”
Algorithmic stability for adaptive data analysis (2015, methodology): “…settings, for example: in multistage inference algorithms where data are preprocessed (say, to select features or restrict to a principal subspace) before the main analysis is performed; in scoring data-based competitions [BH15]; and in the re-use of holdout or test data [DFH + 15c, DFH + 15a].”
Simulation-assisted machine learning (2018, methodology): “hallenge. \Pure data" approaches to this problem are underpowered to detect combinatorially complex interactions and are bound to uncover false correlations despite statistical precautions taken (1). To investigate this setting, we propose a method to integrate simulations, a strong form of pri”
In Search of Lost Domain Generalization (2020, methodology): “As an alternative to limiting the number of queries, we could borrow tools from differential privacy, previously applied to enable multiple re-uses of validation sets in standard supervised learning [Dwork et al., 2015].”
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification (2018, methodology): “In practice, when we have few samples, we can put all of our samples in one batch and use noise-addition techniques to reduce overfitting [6, 27].”
Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle (2015, methodology): “Theorem 7) In this Appendix, we describe how the results in Dwork et al. (2015b) can be used to obtain the forward direction of our characterization without going through a stability argument.”
How Much Does Your Data Exploration Overfit? Controlling Bias via Information Usage (2015, methodology): “Recent work has proposed max-information [19], and its generalization, approximate max-information, as a metric to control the error of a worst-case, adversarial, data analyst.”
Private False Discovery Rate Control (2015, background): “This connection between multiple hypothesis testing and diﬀerential privacy provided the seeds for an exciting recent result showing that diﬀerential privacy protects against false discoveries due to adaptive data analysis [8].”
Discovering Unwarranted Associations in Data-Driven Applications with the FairTest Testing Toolkit (2015, background): “More efficient approaches have been proposed [11], but they all lead to similar systems implications, restrictions, and interfaces.”
Adaptive Learning with Robust Generalization Guarantees (2016, background): “These approaches are very natural, but also can lead to false discovery in the ﬁrst case, and disastrous overﬁtting to the holdout set in the second [DFH + 15c], because traditional generalization is not robust to adaptive composition .”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, background): “These serve as a useful starting point for evaluation, although users should be aware of the danger of overﬁtting to these evaluation sets [Dwork et al., 2015].”