Pitfalls in Machine Learning Research: Reexamining the Development Cycle

By Stella Biderman, Walter J. Scheirer. 2020

28 citations recorded by the source corpus.

Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “As the scale of ML research grows, it becomes increasingly difficult to validate automatically collected and curated datasets (Biderman and Scheirer, 2020; Prabhu and Birhane, 2020; Bender et al., 2021).”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “…and Cao and Daumé III (2020) argue that socially discriminatory biases can be encoded in several stages of the LLM development process (Biderman and Scheirer, 2020), including data sampling, annotation, selection of input representations or model, research design, and how the models are situated…”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, background): “These concerns have been well documented for text corpora (Biderman and Scheirer, 2020; Bender et al., 2021).”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “The absence of community norms like reproducibility guidance [34, 114], documentation standards [98] or statistical signiﬁcance testing [16] makes relying on outcomes-based approaches to evaluate progress even more questionable [13].”
Cut the CARP: Fishing for zero-shot story evaluation (2021, result): “This is key to avoiding overﬁtting due to data leakage, otherwise the model could identify (passage, critique) pairs by matching the quotes (Biderman & Scheirer, 2020; Elangovan et al., 2021).”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Potential issues with such include token repetitions [33, 21, 1], misogyny, pornography, and malignant stereotypes [9, 7], benchmark data contamination [11, 14], spurious correlations [27, 37, 38], diluted robustness due to data mixing [45] and potentially sensitive information [20].”