Lessons from archives: strategies for collecting sociocultural data in machine learning

By Eun Seo Jo, Timnit Gebru. 2019

379 citations recorded by the source corpus.

A growing body of work shows that many problems in fairness, accountability, transparency, and ethics in machine learning systems are rooted in decisions surrounding the data collection and annotation process. In spite of its fundamental nature however, data collection remains an overlooked part of the machine learning (ML) pipeline. In this paper, we argue that a new specialization should be formed within ML that is focused on methodologies for data collection and annotation: efforts that require institutional frameworks and procedures. Specifically for sociocultural data, parallels can be drawn from archives and libraries. Archives are the longest standing communal effort to gather human information and archive scholars have already developed the language and procedures to address and discuss many challenges pertaining to data collection such as consent, power, inclusivity, transparency, and ethics & privacy. We discuss these five key approaches in document collection practices in archives that can inform data collection in sociocultural ML. By showing data collection practices from another field, we encourage ML research to be more cognizant and systematic in data collection and draw from interdisciplinary expertise.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Safety Research

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Improving language models by retrieving from trillions of tokens (2021, background): “Bender et al. (2021); Jo and Gebru (2020) advocate for the importance of better training data curation and documentation.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, neutral): “, 2020), and archival data collection (Jo and Gebru, 2020).”
AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “More generally, dataset development practices within machine learning have been characterized by a “laissez-faire” attitude, rather than a more thoughtful, intentional, and curatorial approach [Jo and Gebru, 2020].”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Jo and Gebru (2020) discuss parallels between creating historical archives and the curation of machine learning datasets including pretraining corpora.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “This requires dedicated and expensive efforts in data curation (Hutchinson et al., 2021; Jo and Gebru, 2020; Kammoun et al., 2022) and novel data pipelines (Denton et al.”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Proposed solutions to dataset bias include new data collection approaches [38, 35], manual data cleanup [74, 75], synthetic data generation [58, 20, 60] – or, in extreme cases, even withdrawing the dataset after insurmountable biases have been identifed [13].”
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023, background): “[5] contend—“too large to document” both technically [5] and methodologically [14].”
Measuring Data (2022, methodology): “Combined with the modern laissez-faire approach to dataset development and usage, which is shallow and limited when it is done at all (Jo and Gebru, 2020), we are currently in the midst of creating, sharing, and training models based on datasets that we know very little about, contributing to…”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Jo and Gebru characterize the resulting culture of dataset development as one that embodies a laissez-faire attitude, which they contrast with the careful and critical curatorial practices of archivists [62].”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “…likely require the community to adopt more mindful data collection practices (Gehman et al., 2020; Bender and Friedman, 2018; Gebru et al., 2018; Jo and Gebru, 2020; Paullada et al., 2020; Bender et al., 2021), establish standardized ﬁltering pipelines for corpora (Roziewski and Stokowiec,…”
Challenges in Automated Debiasing for Toxic Language Detection (2021, background): “First, due to how its training data was selected, GPT-3 has likely not been exposed to many African American English varieties during training (Jo and Gebru, 2020).”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “Data work is heavily under-incentivized, with most attention being paid to algorithmic developments (Jo and Gebru, 2020; Hutchinson et al., 2021; Sambasivan et al., 2021) and publications that focus solely on dataset creation tend to be devalued within traditional peer-review processes…”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “…especially libraries and archives, who have existing roles in curating and describing collections could result in easier access to metadata and data resources provided by these institutions and also support the development of ethical best practices(Jo and Gebru, 2020) and meta-data standardization.”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “In collecting, documenting, and archiving, sociocultural data needed to train and validate machine learning systems, participatory approaches can be critical for representing the “non-elites, the grassroots, the marginalized” in datasets, as outlined by Jo and Gebru [48].”