Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning

By Natã Miccael Barbosa, Mon-Chu Chen. 2019

107 citations recorded by the source corpus.

The increased use of machine learning in recent years led to large volumes of data being manually labeled via crowdsourcing microtasks completed by humans. This brought about dehumanization effects, namely, when task requesters overlook the humans behind the task, leading to issues of ethics (e.g., unfair payment) and amplification of human biases, which are transferred into training data and affect machine learning in the real world. We propose a framework that allocates microtasks considering human factors of workers such as demographics and compensation. We deployed our framework to a popular crowdsourcing platform and conducted experiments with 1,919 workers collecting 160,345 human judgments. By routing microtasks to workers based on demographics and appropriate pay, our framework mitigates biases in the contributor sample and increases the hourly pay given to contributors. We discuss potential extensions and how it can promote transparency in crowdsourcing.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “…it was crowdsourced, and thus susceptible to reﬂecting the gender biases known to be present in crowdworkers’ annotations (Otter-bacher et al., 2018; Barbosa and Chen, 2019), and third, LIGHT’s medieval, fantasy setting might encourage crowdworkers to impart text with their gender biases.”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “as underpayments in crowdsourcing (see [5] for a brief overview).”
Critical Race Theory for HCI (2020, background): “Other forms of algorithmic bias creep in through flawed training data that is itself biased, whether for historical reasons, flawed human labeling, or otherwise [5, 89, 106].”
Evaluating Gender Bias in Natural Language Inference (2021, background): “This has prompted a large area of research attempting to evaluate and mitigate them, either through removal of bias introduction in dataset level (Barbosa & Chen, 2019), or through model architecture (Gonen & Goldberg, 2019), or both (Zhou & Bansal, 2020).”