“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI

By Nithya Sambasivan, Shivani Kapania, H. Highfill, Diana Akrong, Praveen K. Paritosh, Lora Aroyo. 2021

992 citations recorded by the source corpus.

AI models are increasingly applied in high-stakes domains like health and conservation. Data quality carries an elevated significance in high-stakes AI due to its heightened downstream impact, impacting predictions like cancer detection, wildlife poaching, and loan allocations. Paradoxically, data is the most under-valued and de-glamorised aspect of AI. In this paper, we report on data practices in high-stakes AI, from interviews with 53 AI practitioners in India, East and West African countries, and USA. We define, identify, and present empirical evidence on Data Cascades—compounding events causing negative, downstream effects from data issues—triggered by conventional AI/ML practices that undervalue data quality. Data cascades are pervasive (92% prevalence), invisible, delayed, but often avoidable. We discuss HCI opportunities in designing and incentivizing data excellence as a first-class citizen of AI, resulting in safer and more robust systems for all.

Subfields: Artificial Intelligence, Computer Science Applications, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Safety Research

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Similarly, through interviews with 53 AI practicioners, Sambasivan et al. (2021) highlight how data collection choices cascade and amplify through all parts of the development pipeline from training and evaluation to deployment.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Motivation The disconnect between developers and (in)voluntary users of the technology mentioned in Section 2 is particularly apparent in the curation of the datasets that have supported recent large-scale machine learning projects, where intentional “Data work” is generally under-valued (Sambasivan”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “This highlights the general problem with the lack of documentation in NLP datasets of all sizes (Bender and Friedman, 2018; Gebru et al., 2020), and the fact that data work has generally not been a priority in NLP recently (Sambasivan et al., 2021).”
Measuring Data (2022, methodology): “Such research is necessary for enabling better incentives for data, as it is hard to improve something we can not measure” (Sambasivan et al., 2021)”
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, background): “Recent work has drawn attention to various issues caused by the lack of value put on data work in our research community (Sambasivan et al., 2021), and to how prioritizing efﬁciency and technical performance comes at the expense of social considerations for datasets (Scheuerman et al., 2021;…”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “In combination, these concerns reflect what Jo and Gebru(47) describe as a laissez-faire attitude regarding dataset development and the pervasive undervaluation of data work.(48) Rather than collecting and curating datasets with care and intentionality—as is more typical in other data-centric discip”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, background): “Our findings here are in line with recent characterizations of machine learning dataset work as undervalued and deprioritized [58, 95].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “Data is frequently identified as a key source of these failures through its role in “bias-laundering” [40, 51, 54, 119, 125].”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “Second, research addressing dataset choices, creation, and curation, is systematically “under-valued and de-glamorised” [3, 123]16.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “Additionally, the motivation for contributing to this kind of activity may also suffer as a result of a broader undervaluing of datarelated work in NLP and ML (Sambasivan et al., 2021).”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “The recent upswing in participatory tools have varied in approach and point in the machine learning product lifecycle, including: auditing methods for machine learning based interventions [48, 75], public consultation methods such as citizen juries [6, 83] or joint problem formulation [61], informat”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “Despite their importance, work on improving the quality and documentation of datasets has been largely under-valued by the community, with the greater focus given to developing better models and approaches [87].”
Croissant: A Metadata Format for ML-Ready Datasets (2024, neutral): “Data’s prominent role in ML also leads to questions about its responsible use for training and evaluating ML models in areas such as licensing, privacy, biases, and more [3].”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “While a number of relevant works address the setting where annotator labels are available (Sambasivan et al., 2021; Bouguelia et al., 2018; Tanno et al., 2019a,b; Khetan et al., 2018), this paper addresses the general setting where no annotation information is available except the observed noisy lab”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “In a survey of ML projects across India, East and West African countries, and the USA, Sambasivan et al. (2021) analyze compounding events causing negative, downstream effects from data issues, resulting in technical debt 4 over time, and identify insufﬁ-2 cient documentation to be a trigger of thes”
Changing the World by Changing the Data (2021, background): “…sub-groups, their values, and underlying social constructs may also differ across communities: e.g. both in India and US there is discrimination based on skin tone, but in the US context it stands for race, and in India it is a proxy for ethnicity, caste and class (Sambasivan et al., 2021a).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “learning practitioners encounter data cascades, or downstream problems resulting from poor data quality (Sambasivan et al., 2021).”