Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from?

By R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, Jenny Huang. 2019

183 citations recorded by the source corpus.

Many machine learning projects for new application areas involve teams of humans who label data for a particular purpose, from hiring crowdworkers to the paper's authors labeling the data themselves. Such a task is quite similar to (or a form of) structured content analysis, which is a longstanding methodology in the social sciences and humanities, with many established best practices. In this paper, we investigate to what extent a sample of machine learning application papers in social computing --- specifically papers from ArXiv and traditional publications performing an ML classification task on Twitter data --- give specific details about whether such best practices were followed. Our team conducted multiple rounds of structured content analysis of each paper, making determinations such as: Does the paper report who the labelers were, what their qualifications were, whether they independently labeled the same items, whether inter-rater reliability metrics were disclosed, what level of training and/or instructions were given to labelers, whether compensation for crowdworkers is disclosed, and if the training data is publicly available. We find a wide divergence in whether such practices were followed and documented. Much of machine learning research and education focuses on what is done once a "gold standard" of training data is available, but we discuss issues around the equally-important aspect of whether such data is reliable in the first place.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, AI Ethics, Fairness & Society, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Sociology and Political Science

In AI Ethics, Fairness & Society

In Social Sciences & Humanities

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “What are the impacts of annotation guidelines, annotator assumptions and perceptions (Olteanu et al., 2019; Sap et al., 2019; Geiger et al., 2020), and annotation aggregation processes (Pavlick and Kwiatkowski, 2019)?”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Finally, critical aspects of crowdwork are often left poorly documented, or entirely undocumented [88].”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “For example, relatively little attention is given or documented about annotator positionality—how annotator social identity shapes their understanding of the world [18, 48].”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “A recent study of publications that leverage Twitter data found data decisions were heavily under-specified and inconsistent across publications.(60) Scheuerman et al.”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Geiger et al., meanwhile, do not provide a formal diagnostic or checklist for the construction of dataset documentation but form one implicitly by analyzing a set of papers focused on social computing [44].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “ew process of top-tier venues [67] and—perhaps due to the imperative to squeeze dataset details the methods-oriented papers— critical dataset design decisions are frequently left underspecified (e.g. [53]). Despite the foundational role data place in machine learning research and development, data o”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “In an analogous project, (Geiger et al., 2020) analyzed the data collection and annotation practices of over a hundred social computing articles analyzing Twitter data and found a lack of consistent standardized practices of documentation.”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “standards [26, 38] for data enrichment ormoderation tasksmeans that these contributors often fail to be discussedwhen publishing the final artefacts or protected by conventional research participant standards (e.”
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, background): “Research into the disciplinary 35 norms of dataset development have revealed troubling practices around dataset development and 36 dissemination, like unstandardized documentation and maintenance practices [10, 11, 12].”
ORES (2019, methodology): “ystem, but had different values, approaches, and procedures. 5.1.1 Trace extraction and manual labeling. Labeled data is the heart of any model, as ML can only be as good as the data used to train it [39]. It is through focusing on labeled observations that we help requesters understand and negotiat”
A Hierarchy of Limitations in Machine Learning (2020, background): “This is especially true when, as documented by Geiger et al. (2020) for social computing, machine learning researchers frequently report doing the labeling themselves, or else do not even report where labels come from.”
Between Subjectivity and Imposition (2020, background): “Moreover, the structures, decisions, actors, and frameworks which shape data annotation should be made explicit [39, 72].”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “In a recent survey of machine learning papers (including NLP ones), Geiger et al. (2020) reported a wide divergence in the level of documentation about methodological practices in human annotation.”
One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision (2021, background): “Geiger et.al [16] find significant variability in the quality of human annotations in a case study of papers on tweet classification.”
Censorship of Online Encyclopedias: Implications for NLP Models (2021, background): “More generally, research in machine learning has been criticized for not paying enough attention to the origin of training datasets and the social processes that generate them [15].”
Artsheets for Art Datasets (2021, background): “Unstandardized dataset documentation,and maintenance frameworks have been identiﬁed as a key barrier to reproducibility and accountability [21, 22, 23, 15].”