Toward Annotator Group Bias in Crowdsourcing
By Haochen Liu, J. Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, Jiliang Tang. 2021
20 citations recorded by the source corpus.
Crowdsourcing has emerged as a popular approach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating individual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same demographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilistic graphical framework GroupAnno to capture annotator group bias with an extended Expectation Maximization (EM) algorithm. We conduct experiments on both synthetic and real-world datasets. Experimental results demonstrate the effectiveness of our model in modeling annotator group bias in label aggregation and model learning over competitive baselines.
Subfields: Evaluation & Benchmarks, AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Matt Gardner
- Pontus Stenetorp
- Pradeep Dasigi
- Yuxiang Wu
- Jonathan Berant
- Tushar Khot
- Ashish Sabharwal
- Hannaneh Hajishirzi
Citations in the corpus
- Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets (2022, methodology): “, 2019) or group-level (Liu et al., 2021) annotator biases.”