Controlling Bias in Adaptive Data Analysis Using Information Theory

By Daniel Russo, James Y. Zou. 2015

254 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Generalization for Adaptively-chosen Estimators via Stable Median (2017, methodology): “Generalization guarantees in the adaptive setting can also be derived via other approaches such as bounding the approximate max-information or mutual information between the choice of estimator and the data (Dwork et al., 2015a; Rogers et al., 2016; Russo and Zou, 2016) or using other notions of sta”
A Minimax Theory for Adaptive Data Analysis (2016, methodology): “…studied settings as special cases; • we present the first rate-optimal minimax lower bound for adaptive data analysis in the setting studied in Russo & Zou (2015); • we present a new proof of the upper bound which yields a sharper constant and a transparent understanding of the least…”
Data Cleaning: Overview and Emerging Challenges (2016, background): “Adaptive Data Analysis: In a recent development, the statistics community has studied some of these problems in a ﬁeld called “Adaptive Data Analysis", and we will also overview some of that work [66, 29].”
Max-Information, Differential Privacy, and Post-selection Hypothesis Testing (2016, methodology): “In the full version , we demonstrate that if a bound on the mutual information between the dataset and the output of the selection procedure is known, then it is possible to substantially improve on the p-value correction function given by [10] by instead using the mutual information bound to prove ”
Calibrating Noise to Variance in Adaptive Data Analysis (2017, methodology): “Another known implication of bounded mutual information is that any event that would happen with sufficiently low probability on fresh data will still happen with low probability (Russo and Zou, 2016; Rogers et al., 2016).”
Why adaptively collected data have negative bias and how to correct for it (2017, result): “There is a related line of recent work [7] [17] in adaptive data analysis that is complementary to ours.”
Adaptive Learning with Robust Generalization Guarantees (2016, unknown): “A recent line of work (Dwork et al., 2015c,a; Bassily et al., 2016; Russo and Zou, 2016) has studied algorithmic conditions that guarantee the sort of robust generalization guarantees we study in this paper, suitable for adaptive data analysis.”
Generalization Error Bounds for Noisy, Iterative Algorithms (2018, methodology): “Generalization error bounds The idea of bounding generalization error by the mutual information I(W ;S) between the input and output of an ERM algorithm was first proposed by Russo and Zou [4] and further investigated by Xu and Raginsky [5].”