The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

By Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, Kasia Chmielinski. 2018

325 citations recorded by the source corpus.

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

Subfields: Public Health, Environmental and Occupational Health, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Public Health, Environmental and Occupational Health

In Evaluation & Benchmarks

Citations in the corpus

Model Cards for Model Reporting (2018, methodology): “Model cards (for model reporting) are complements to “Datasheets for Datasets” [21] and similar recently proposed documentation paradigms [3, 28] that report details of”
Datasheets for datasets (2018, background): “We are heartened to see other groups working on projects similar to datasheets for datasets (Holland et al., 2018).”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, background): “CrowdWorkSheets complements and extends dataset development and documentation frameworks that have previously been developed in service of transparency, accountability, and reproducibility [6, 9, 17, 26, 27, 40, 42], but focuses specifically on unique considerations relating to crowdsourced dataset ”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “, 2019), and data exploration (Holland et al., 2018), as well as generalist work such as (Gebru et al.”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “For instance, Gebru et al., Bender and Friedman, and Holland et al. have all put forth data reporting frameworks for improving dataset documentation [10, 43, 57].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “Transparent reporting of machine learning datasets [16, 52, 70] and the models trained on those data [99] enables third party agents, including potential users, to make informed decisions.”
A Survey on Bias and Fairness in Machine Learning (2019, background): “Authors in [57] also propose having labels, just like nutrition labels on food, in order to better categorize each data for each task.”
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “sampling and measurement) [44, 54] enables modeling that appropriately accounts for the specific conditions under which data has been collected (Chapter 8 of [46], [110, 83]).”
Increasing Trust in AI Services through Supplier's Declarations of Conformity (2018, background): “Holland et al. outline the dataset nutrition label , a diagnostic framework that provides a concise yet robust and standardized view of the core components of a dataset [12].”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “The recent upswing in participatory tools have varied in approach and point in the machine learning product lifecycle, including: auditing methods for machine learning based interventions [48, 75], public consultation methods such as citizen juries [6, 83] or joint problem formulation [61], informat”
Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus (2021, methodology): “…address “documentation debt” for the widely-used BookCorpus dataset, following a growing body of work suggesting that inﬂuential datasets need more careful documentation (i.e. “dataset nutrition labels” [18, 19, 34], “data statements” [1], “dataset disclosure forms” [33], or “datasheets” [2, 17]).”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, methodology): “Additional metadata include data provenance, which tracks where and when a piece of data comes from, and the processes and methodology by which it was produced 41 .”
ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles (2019, background): “…industry [4] or nutrition labels from the food industry [7], the suggested templates vary widely in length and appearance, ranging from a single concise page of succinct statements [11] or a set of symbols and visualizations [7, 10] to upwards of 10 pages of detailed prose and graphs [4].”