On Releasing Annotator-Level Labels and Information in Datasets

By Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, Mark Díaz. 2021

191 citations recorded by the source corpus.

A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single “ground truth” label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Sociology and Political Science

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “Prabhakaran et al. (2021) find that aggregating annota- 15See, for example, Ehud Reiter’s summary of a panel on NLG in industry at INLG 2021. tor responses results in under-representation of groups of annotators’ opinions, and they recommend releasing annotator-level annotations and collecting…”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Annotators often have different perceptions of concepts like race or are influenced by their own lived experience [234] when categorizing protected categories [187].”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “We retain individual annotations in the dataset to capture potential differences in annotator behavior owing to their socio-cultural background and lived experiences (Prabhakaran et al., 2021).”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “Additionally, re-questers should be mindful of potential diﬀerences between legal minimum wages and a living wage [21].”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “Prior work has extensively shown links between someone’s gender, political leaning, and race affects how likely they are to perceive or notice harmful speech or racism (Cowan et al., 2002; Norton and Sommers, 2011; Carter and Murphy, 2015; Prabhakaran et al., 2021).”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “An adverse effect of majority vote in such cases is limiting representation of minority perspectives in data (Prabhakaran et al., 2021), potentially reinforcing societal disparities and harms.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “As a result, when preferences differ, the majority wins, potentially disadvantaging under-represented groups [Prabhakaran et al., 2021, Feffer et al., 2023, Kirk et al., 2023].”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “For the purposes of ML model evaluations, disaggregating labels on test data according to the cultural and socio-demographic standpoints of their annotators enables more nuanced disaggregated evaluation statistics [131].”
The Ghost in the Machine has an American accent: value conflict in GPT-3 (2022, background): “engineers, the writers of the new targeted dataset, and critically the ‘evaluators’ of the generated output[68].”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “all annotations with each dataset, even disagreeing ones, allows training models that generalise better (Plank et al., 2014; Pavlick and Kwiatkowski, 2019; Prabhakaran et al., 2021; Gordon et al., 2022).”