Annotating Social Media Data From Vulnerable Populations: Evaluating Disagreement Between Domain Experts and Graduate Student Annotators

By Desmond Upton Patton, Philipp Blandfort, William R. Frey, M. Gaskell, Svebor Karaman. 2019

34 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “This is problematic as there are known diﬀerences in how non-experts and experts interact with and validate systems outputs (White et al., 2009; Patton et al., 2019), particularly for critical application domains such as health.”
On Releasing Annotator-Level Labels and Information in Datasets (2021, background): “…Annotators’ socio-demographic factors, moral values, and lived experiences often inﬂuence their interpretations of language, especially in subjective tasks such as identifying political stances (Luo et al., 2020), sentiment (Díaz et al., 2018), and on-line abuse (Waseem, 2016; Patton et al., 2019).”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “For instance, in cases where annotators with different moral beliefs systemically produce different labels (Waseem, 2016; Díaz, 2020; Patton et al., 2019), one could use the multi-task approach to have a single global model that can adjust predictions to be conditioned on different value systems.”
A Hierarchy of Limitations in Machine Learning (2020, background): “Or, if there are untrained annotators applying their own heuristic understandings to produce labels, the results may be meaningless and useless (Patton et al., 2019a) regardless of how how high the “accuracy” is of a model to recover those annotations.”