Leakage in data mining: formulation, detection, and avoidance

By Shachar Kaufman, Saharon Rosset, Claudia Perlich. 2011

672 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

Citations in the corpus

Anchors: High-Precision Model-Agnostic Explanations (2018, background): “Relying only on validation accuracy has many well studied problems, as practitioners consistently overestimate their model’s accuracy (Patel et al. 2008), propagate feedback loops (Sculley et al. 2015), or fail to notice data leaks (Kaufman, Rosset, and Perlich 2011).”
Auditing black-box models for indirect influence (2016, background): “Another related framework is that of leakage in data mining [15], which investigates whether the methodology of the mining process is allowing information to leak from test data into the model.”
Interpreting tree ensembles with inTrees (2014, neutral): “Keywords: decision tree; rule extraction; rule-based learner; random forest; boosted trees.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “At times, data features accidentally contain direct information about the target variable in a way that makes the learning problem redundant [70].”
Fairness-aware Learning through Regularization Approach (2011, background): “Fairness in machine learning can be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [13].”
Auditing Black-box Models by Obscuring Features (2016, background): “ssiﬁcation with sensitive information from the input. In our setting, the “sensitive” information must be hidden from the classiﬁer itself. Another related framework is that of leakage in data mining [15], which investigates whether the methodology of the mining process is allowing information to le”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2021, result): “Similar is the case for data leakage, which can mislead the model learning and testing [49].”
Fairness-Aware Classifier with Prejudice Remover Regularizer (2012, background): “Fairness in data mining can be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [19].”
Explaining Explanations to Society (2019, background): “In [13], one classiﬁer learned the resolution of the scanner camera, therefore predicting cancerous images from a high resolution very accurately.”
Interpretable Machine Learning in Healthcare (2018, background): “testing [14], however explanations can be used to interrogate and rectify models when such problems surface.”
Correlated daily time series and forecasting in the M4 competition (2020, methodology): “In such case using the end of y to forecast the continuation of y would be using data from the future (forecast earlier dates using data from later dates), which is of course not doable in practical applications and demonstrates data leakage (Kaufman et al., 2012) in the M4 data.”
Leakage and the reproducibility crisis in machine-learning-based science (2023, background): “Patterns 4, 100804, September 8, 2023 7 The main findings of each of these papers are invalid due to various forms of data leakage: Muchlinski et al.75 impute the training and test data together, Colaresi and Mahmood76 and Wang77 incorrectly reuse an imputed dataset, and Kaufman et al.78 use proxies”
Federated learning for predicting clinical outcomes in patients with COVID-19 (2021, background): “Recent works on privacy attacks within the FL setting have raised concerns on data leakage during model training 59 .”