CDAS: A Crowdsourcing Data Analytics System

By Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, Meihui Zhang. Proceedings of the VLDB Endowment, 2012

300 citations recorded by the source corpus.

Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively. To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/71b76a4de9b1364be2905278b69e0912c5f70b3a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
Foundations of Crowd Data Sourcing (2015, unknown): “, [3, 6, 7, 9, 12, 13, 14, 15, 17, 19, 24, 26, 28].”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “There are three research topics in crowdsourcing related to our work: task assignment [4, 6, 13, 17, 26, 31, 46], truth inference [12,16,19,33,45], and active learning (from crowds) [18, 24, 28, 43].”
Crowdsourced Data Management: A Survey (2016, background): “As a single worker may be biased for some tasks, most existing works [15], [22], [58], [69], [74], [138], [139] often employ a quality-control strategy, called answer aggregation (or voting strategy).”
Corleone: hands-off crowdsourcing for entity matching (2014, background): “Many other works crowdsource joins [19], find the maximal value [11], collect data [28], match schemas [21], and perform data mining [1] and analytics [18].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, methodology): “These papers consider increasingly expressive models for this estimation problem, including worker bias [100], difficulty of tasks and worker expertise [193, 157, 106, 108], adversarial behavior [156], and online evaluation of workers [191, 128, 155].”
Crowdsourcing large scale wrapper inference (2014, unknown): “, [12,29,35].”
Decision Tree Design for Classification in Crowdsourcing Systems (2018, background): “These problems have also been extended to budget-constrained environments to improve classification performance [10]–[12].”