Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI

By Mahima Pushkarna, Andrew Zaldivar, Oddur Kjartansson. 2022

352 citations recorded by the source corpus.

As research and industry moves towards large-scale models capable of numerous downstream tasks, the complexity of understanding multi-modal datasets that give nuance to models rapidly increases. A clear and thorough understanding of a dataset’s origins, development, intent, ethical considerations and evolution becomes a necessary step for the responsible and informed deployment of models, especially those in people-facing contexts and high-risk domains. However, the burden of this understanding often falls on the intelligibility, conciseness, and comprehensiveness of the documentation. It requires consistency and comparability across the documentation of all datasets involved, and as such documentation must be treated as a user-centric product in and of itself. In this paper, we propose Data Cards for fostering transparent, purposeful and human-centered documentation of datasets within the practical contexts of industry and research. Data Cards are structured summaries of essential facts about various aspects of ML datasets needed by stakeholders across a dataset’s lifecycle for responsible AI development. These summaries provide explanations of processes and rationales that shape the data and consequently the models—such as upstream sources, data collection and annotation methods; training and evaluation methods, intended use; or decisions affecting model performance. We also present frameworks that ground Data Cards in real-world utility and human-centricity. Using two case studies, we report on desirable characteristics that support adoption across domains, organizational structures, and audience groups. Finally, we present lessons learned from deploying over 20 Data Cards.x

Subfields: Artificial Intelligence, Management Information Systems, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “There are many research projects focusing on developing evaluation infrastructure, from human evaluation datasheets (Shimorina & Belz, 2021), to NLG-specific data cards with interactive collection tools (McMillan-Major et al., 2021; Pushkarna et al., 2022), metrics frameworks (Gehrmann et al.”
Scaling Instruction-Finetuned Language Models (2022, methodology): “We present the data card, following the format proposed by Pushkarna et al. (2021).”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “[47] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson.”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “CrowdWorkSheets complements and extends dataset development and documentation frameworks that have previously been developed in service of transparency, accountability, and reproducibility [6, 9, 17, 26, 27, 40, 42], but focuses specifically on unique considerations relating to crowdsourced dataset ”
PaLM 2 Technical Report (2023, background): “Dataset analysis and transparency artifacts are a key part of Responsible AI practices (Bender & Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021; Pushkarna et al., 2022).”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “Examples and statistics for the WebLI corpus and a complete datasheet (Pushkarna et al., 2022) are shown in Appendix B (Figure 4) and G. Training mixture To accommodate diverse tasks in the image-language space, we train PaLI using a mixture of eight pre-training tasks.”
Auditing large language models: a three-layered approach (2023, background): “, interactive model cards [183] and information about the initial training dataset [184], [185], to help downstream developers adapt the model appropriately.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “Associated datasheets [17] and data cards [41] for EgoSchema are provided in the supplementary .”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “We analyzed the existing template and the resulting data cards under the dimensions provided in the data card playbook (Pushkarna et al., 2022) and”
Masader Plus: A New Interface for Exploring +500 Arabic NLP Datasets (2022, background): “Data Cards are defined as structured summaries of essential facts about various aspects of ML datasets needed by stakeholders across a dataset’s life-cycle for responsible AI development (Pushkarna et al., 2022).”