Datasheets for datasets

By Timnit Gebru, Jamie H. Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé Iii, Kate Crawford. 2018

3,178 citations recorded by the source corpus.

Documentation to facilitate communication between dataset creators and consumers.

Subfields: Artificial Intelligence, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, background): “(and emphasise) the presence of human bias. Using such sensational “party tricks" (Gonen and Goldberg 2019) is harmful, as they get easily propagated both in science itself (Jha and Mamidi 2017; Gebru et al. 2018; Mohammad et al. 2018), even outside NLP and AI (McQuillan2018), as well as in pop”
Model Cards for Model Reporting (2018, methodology): “Each model card could be accompanied with Datasheets [21], Nutrition Labels”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…the choices of dataset creators, it is imperative to report the curation decisions alongside limitations of datasets in structured format to allow for a better interpretation and contextualization of performance results (Bender and Friedman, 2018; Gebru et al., 2021; McMillan-Major et al., 2021).”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “Part of the challenge of addressing this lack of context is proper documentation for these datasets, which is often underdeveloped [Gebru et al., 2020, Bender and Friedman, 2018] and the devaluation of the data work [Paullada et al., 2020, Sambasivan et al., 2021, Hutchinson et al., 2020, Jo and…”
Deduplicating Training Data Makes Language Models Better (2021, background): “While recent work focused on documenting the potential harms that could arise from problematic datasets (Bender and Friedman, 2018; Gebru et al., 2020), less work has been done to quantitatively analyze properties of real language modelling datasets, like Dodge et al. (2021a) has done for C4.”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “Paullada et al. (2020), Bender and Fried-man (2018), and Gebru et al. (2018) provide more comprehensive surveys and frameworks that focus on aspects of data creation and management that could lead to biases, and we refer readers to their works for more discussion.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “EN dataset. structured, task-speciﬁc NLP datasets, best practices have emerged around documenting the collection process, composition, intended uses, and other characteristics (Bender and Friedman, 2018; Gebru et al., 2018; Hutchinson et al., 2021).”
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, methodology): “Existing documentation approaches, like datasheets for datasets (Gebru et al. 2018) and model cards (Mitchell et al.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “cards (Bender and Friedman, 2018; Gebru et al., 2018) we developed and for which we release a template and guide.”
Measurement and Fairness (2019, background): “Much of the work in the space of fairness also relies on metrics and other approaches that assume access to attributes, such as race or gender (for instance, structured transparency documents [26]).”
Toward Gender-Inclusive Coreference Resolution (2019, background): “Task definitions for linguistic annotations (like coreference) tend, in NLP, to be described in annotation guidelines (or, more recently, in datasheets or data statements [Bender and Friedman 2018; Gebru et al. 2018]).”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “We follow the recommendations of Gebru et al. (2021) and provide a data card for the dataset used to train the OPT models.”
Predictability and Surprise in Large Generative Models (2022, methodology): “, through the use of model cards [48] and data sheets [26] to document the capabilities, drawbacks, and other salient details of models).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[30] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.”