Efficient Learning for Crowdsourced Regression

By Jungseul Ok, Sewoong Oh, Jinwoo Shin, Yunhun Jang, Yung Yi. 2017

3 citations recorded by the source corpus.

Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, one of the fundamental questions is how to identify more reliable workers and exploit this heterogeneity to infer the true answers. Despite significant research efforts for classification tasks with discrete answers, little attention has been paid to regression tasks where the answers take continuous values. We consider the task of recovering the position of target objects, and introduce a new probabilistic model capturing the heterogeneity of the workers. We propose the belief propagation (BP) algorithm for inferring the positions and prove that it achieves optimal mean squared error by comparing its performance to that of an oracle estimator. Our experimental results on synthetic datasets confirm our theoretical predictions. We further emulate a crowdsourcing system using PASCAL visual object classes datasets and show that de-noising the crowdsourced data using BP can significantly improve the performance for the downstream vision task.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuji Roh
2. Steven Euijong Whang
3. Geon Heo

In Computer Science Applications

1. Geon Heo
2. Yuji Roh
3. Steven Euijong Whang

Citations in the corpus

• A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “[114] propose a probabilistic model for crowdsourced regression for identifying reliable workers and taking into account the heterogeneity of multiple workers to infer the answers accurately.”