CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
By Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, Emily L. Denton. 2022
111 citations recorded by the source corpus.
Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators’ lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.
Subfields: Artificial Intelligence, Computer Science Applications, Safety Research, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Noah A. Smith
- Jianfeng Gao
- Percy Liang
- Yejin Choi
In Computer Science Applications
- Yi Tay
- Mostafa Dehghani
- Vinodkumar Prabhakaran
- Irene Solaiman
- Ben Hutchinson
- Nan Du
- Mark Díaz
- Alexandra Sasha Luccioni
In Safety Research
- Sunipa Dev
- Irene Solaiman
- Ben Hutchinson
- Vinodkumar Prabhakaran
- Frederick Liu
- Elizabeth Clark
- Yi Tay
- Thibault Sellam
In Evaluation & Benchmarks
- Sebastian Ruder
- Sebastian Gehrmann
- Elizabeth Clark
- Thibault Sellam
- Colin Cherry
- Orhan Firat
- Melvin Johnson
- Markus Freitag
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “, 2021) or CrowdWorkSheets (Dı́az et al., 2022) and by contributing and using projects that standardize parts of the process (e.”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Concurrently, researchers and developers should document the role of crowdwork in all dataset development undertaken during generative AI systems development, e.g. using frameworks like CrowdWorkSheets [70] and sections 3.3 and 3.4 in Datasheets for Datasets [86].”
- PaLM 2 Technical Report (2023, methodology): “This dataset consists of English comments annotated by human raters, and we include a CrowdWorksheet (Diaz et al., 2022) describing characteristics of the data collection process in Appendix D.”
- Cultural Incongruencies in Artificial Intelligence (2022, background): “For instance, human raters form a major part of AI workforce, but they are often deemed interchangeable without accounting for the diverse socio-cultural perspectives they bring to the rating task(s) [7].”