Bringing the People Back In: Contesting Benchmark Machine Learning Datasets

By Emily L. Denton, A. Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, Morgan Klaus Scheuerman. 2020

102 citations recorded by the source corpus.

In response to algorithmic unfairness embedded in sociotechnical systems, significant attention has been focused on the contents of machine learning datasets which have revealed biases towards white, cisgender, male, and Western data subjects. In contrast, comparatively less attention has been paid to the histories, values, and norms embedded in such datasets. In this work, we outline a research program - a genealogy of machine learning data - for investigating how and why these datasets have been created, what and whose values influence the choices of data to collect, the contextual and contingent conditions of their creation. We describe the ways in which benchmark datasets in machine learning operate as infrastructure and pose four research questions for these datasets. This interrogation forces us to "bring the people back in" by aiding us in understanding the labor embedded in dataset construction, and thereby presenting new avenues of contestation for other researchers encountering the data.

Subfields: Computer Science Applications, Safety Research, Evaluation & Benchmarks, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

In Evaluation & Benchmarks

In Human-Computer Interaction

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “…reexamination of the relationship between NLP systems and their end users, using methods from humancentered design, such as value-sensitive (Friedman et al., 2008) or participatory design (Sanders, 2002; DiSalvo et al., 2012; Denton et al., 2020), and archival data collection (Jo and Gebru, 2020).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “The limited analysis delegates the responsibility to ensure that all included datasets have been collected fairly to the creators of the benchmark (Denton et al., 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “This requires dedicated efforts in data curation (Hutchinson et al., 2021; Jo and Gebru, 2020; Kammoun et al., 2022) and novel data pipelines (Denton et al., 2020).”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “To avoid harmful outcomes we must address how control over the rules is decided, whether affected parties share in this control, and whether they have visibility into what rules are in effect; considerations analogous to those raised by Denton et al. (2020) for datasets.”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “For example, [11] discuss how the mere inclusion of marginalized groups within a dataset, without sufficient attention to broader considerations of data capture and use, can operate as a form of “predatory inclusion”4.”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “The codebook is part of a larger research project aimed at understanding the genealogy of datasets [28].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “Engineering artefacts are imbued with politics via their roles in sociopolitical systems [57, 147], and so too are ML datasets inseparable from politics [36, 41, 113].”
PaLM 2 Technical Report (2023, methodology): “This rubric evolved from literature review on evaluation (Rauh et al., 2022; Dev et al., 2021b; Bowman & Dahl, 2021; Sambasivan et al., 2021; Paullada et al., 2021; Rodriguez et al., 2021; Schlangen, 2020; Denton et al., 2020; Selbst et al., 2019; Jacobs & Wallach, 2021; Tomasev et al., 2021; Welty ”
Taxonomy of Risks posed by Language Models (2022, background): “Curating training data can also help to make LMs fairer [45, 84, 94].”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “standards [26, 38] for data enrichment ormoderation tasksmeans that these contributors often fail to be discussedwhen publishing the final artefacts or protected by conventional research participant standards (e.”
The Benchmark Lottery (2021, background): “Denton et al. [2020] look at the dataset construction process with respect to the concerns along the ethical and political dimensions of what has been taken for granted, and discuss how thinking about data within a dataset must be holistic, future-looking, and aligned with ethical principles and…”
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, background): “Although we have limited the scope of our work to the contents of visual datasets itself, there are much broader questions of fairness and justice to be considered regarding the role that datasets play (Denton et al., 2020; Paullada et al., 2020; Scheuerman et al., 2021; Peng et al., 2021).”
Epistemic values in feature importance methods: Lessons from feminist epistemology (2021, background): “In the broader context of ML research in general, others have warned of the pitfalls of having benchmarks that determine what “success” or “progress” is [19, 23].”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, background): “In other cases, it may be better to develop entirely new benchmarks that are more representative and better suited to the task at hand [2, 14, 27].”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, unknown): “This has been built upon by further scholarship around the ethical and political dimensions of dataset curation and construction (Prabhu and Birhane 2021; Denton et al. 2020) and the values that are embedded in datasets (Scheuerman, Hanna, and Denton 2021).”
Detecting Independent Pronoun Bias with Partially-Synthetic Data Generation (2020, background): “Our linguistic understanding of (neo)pronouns and inclusive NLP development are areas of ongoing research (Ackerman, 2019; Bradley et al., 2019; Cao and Daum´e III, 2020; Conrod, 2020; Denton et al., 2020; Mitchell et al., 2020).”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “…that participation can enable AI systems to better reflect the values, preferences, and needs of users and other impacted stakeholders, or more broadly, that participation will empower stakeholders in shaping the design of AI systems [3, 7, 19, 26, 33, 51, 61, 118, 137, 143, 172, 189, 198–200].”