Data and its (dis)contents: A survey of dataset development and use in machine learning research

By Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily L. Denton, A. Hanna. 2020

667 citations recorded by the source corpus.

Summary In this work, we survey a breadth of literature that has revealed the limitations of predominant practices for dataset collection and use in the field of machine learning. We cover studies that critically review the design and development of datasets with a focus on negative societal impacts and poor outcomes for system performance. We also cover approaches to filtering and augmenting data and modeling techniques aimed at mitigating the impact of bias in datasets. Finally, we discuss works that have studied data practices, cultures, and disciplinary norms and discuss implications for the legal, ethical, and functional challenges the field continues to face. Based on these findings, we advocate for the use of both qualitative and quantitative approaches to more carefully document and analyze datasets during the creation and usage phases.

Subfields: Artificial Intelligence, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…English (AAVE), Blodgett et al. (2020) describe multiple studies showing a drop in performance on popular NLU tasks when applied to 16We point to Paullada et al. (2020) for a more in-depth survey of general issues in data creation, including those of benchmarking and data maintenance practices,…”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “…challenge of addressing this lack of context is proper documentation for these datasets, which is often underdeveloped [Gebru et al., 2020, Bender and Friedman, 2018] and the devaluation of the data work [Paullada et al., 2020, Sambasivan et al., 2021, Hutchinson et al., 2020, Jo and Gebru, 2020].”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “Paullada et al. (2020), Bender and Fried-man (2018), and Gebru et al. (2018) provide more comprehensive surveys and frameworks that focus on aspects of data creation and management that could lead to biases, and we refer readers to their works for more discussion.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Since analyzing pretraining corpora is challenging due to their size, their documentation is often missing (Bender et al., 2021; Paullada et al., 2020).”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “Therefore, it is critical to uncover the explicit and implicit assumptions that these benchmarks carry and the incentives to which they may give rise (Paullada et al., 2020).”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, background): “[26] describe the need for datasets to be carefully collected in their original context so they are not only representative, but also respect and behave appropriately toward those from whom we collect data.”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Along with many ethical concerns [56, 66], these dataset biases are problematic because they can propagate into models [15, 17].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “For example, public and academic discourse regarding appropriate use of public data has raised concerns regarding data subject awareness and consent [24, 18, 63, 45].”
Foundation Models and Fair Use (2023, background): “The recent literature on data governance and curation discusses fair use in machine learning (Jernite et al., 2022; Paullada et al., 2021; Ganguli et al., 2022).”
Measuring Data (2022, background): “, 2018; Bommasani and Cardie, 2020a); “statistical tendencies”, described as “present” in datasets (Meister and Cotterell, 2021a); “statistical properties” to surface issues with dataset contents (Paullada et al., 2021), and “bias measurements” to quantify dataset bias (Aka et al.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “…of large scale computer vision datasets contains various broad problems including curation biases, inclusion of problematic content in the images, the questionable approaches of associating these images with offensive and non-imageable labels, as well as the gradual erosion of privacy [4, 5].”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Paullada et al. discuss how dataset culture within machine learning prioritizes speed for the achievement of algorithmic performance on a fixed set of benchmarks with little regard to the implications of data reuse, data management, and legal issues [85].”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, methodology): “Very recent research — most published within the last year — has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “…community to adopt more mindful data collection practices (Gehman et al., 2020; Bender and Friedman, 2018; Gebru et al., 2018; Jo and Gebru, 2020; Paullada et al., 2020; Bender et al., 2021), establish standardized ﬁltering pipelines for corpora (Roziewski and Stokowiec, 2016; Ortiz Suarez et…”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “Many of these failures have been the direct result of underrepresentation, misrepresentation, or the complete lack of representation of these groups in the data upon which these systems are built (Paullada et al., 2020).”
A Study of Face Obfuscation in ImageNet (2021, background): “But recently they have been called out for scrutiny (Paullada et al., 2020), especially regarding the presence of people.”