A Survey of Crowdsourcing Systems

By Man-Ching Yuen, Irwin King, K. Leung. 2011

441 citations recorded by the source corpus.

Crowd sourcing is evolving as a distributed problem-solving and business production model in recent years. In crowd sourcing paradigm, tasks are distributed to networked people to complete such that a company's production cost can be greatly reduced. In 2003, Luis von Ahn and his colleagues pioneered the concept of "human computation", which utilizes human abilities to perform computation tasks that are difficult for computers to process. Later, the term "crowdsourcing" was coined by Jeff Howe in 2006. Since then, a lot of work in crowd sourcing has focused on different aspects of crowd sourcing, such as computational techniques and performance analysis. In this paper, we give a survey on the literature on crowd sourcing which are categorized according to their applications, algorithms, performances and datasets. This paper provides a structured view of the research on crowd sourcing to date.

Subfields: Computer Science Applications, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “Other forms include crowdsourced labels [37, 50], rules and heuristics for labeling data [39, 52], and others [29, 30, 30, 46, 51].”
Are We Done with MMLU? (2024, neutral): “…from biases in the framing of the task [26]; noisy annotations [27]; web-crawls [28, 29, 30]; automated labelling processes such as crowdsourcing [31] — where annotators may be optimizing for their own incentives; human errors [32] — e.g., lack of expertise on a given topic; or programmatic weak…”
SoK: The Evolution of Sybil Defense via Social Networks (2013, background): “not just because it results in a goal that, unlike its alternative, is attainable, but because (1) the guarantees it provides are in practice what nodes that engage in crowd-sourcing [47]”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[145] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung.”
Classification in the Presence of Label Noise: A Survey (2014, background): “In addition, since collecting reliable labels is a time consuming and costly task, there is an increasing interest in using cheap, easy-to-get labels from nonexpert using frameworks like e.g. the Amazon Mechanical Turk 1 [49]–[52].”
A survey of the use of crowdsourcing in software engineering (2017, background): “The use of such an undefined workforce inherently1690 raises quality questions for crowdsourcing in general (Ipeirotis et al., 2010; Yuen et al., 2011; Allahbakhsh et al., 2013) as well as Crowdsourced Software Engineering (Stol and Fitzgerald, 2014c; Li et al., 2013; LaToza et al., 2013a;…”
Snorkel: rapid training data creation with weak supervision (2019, background): “Other forms include crowdsourced labels [41,63], rules and heuristics for labeling data [47,65], and others [33,34,34,55,64].”
Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary (2017, methodology): “One approach to this problem is to use crowd sourcing infrastructure like Amazon Mechanical Turk, in combination with techniques for reducing the variance of the evaluation [Yuen et al. , 2011].”
CrowdBC: A Blockchain-Based Decentralized Framework for Crowdsourcing (2019, background): “We refer readers to a comprehensive survey on crowdsourcing for more information [16], [17], [18], [19], [20].”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Traditional methods often only provide initial guidelines to annotators [252].”
A Survey on Programmatic Weak Supervision (2022, background): “Crowd-sourcing is the classic and well-studied approach of obtaining less accurate label annotations from non-expert contributors with lower annotation cost [ Dawid and Skene, 1979; Yuen et al. , 2011 ] .”
Mobile Crowd Sensing and Computing (2015, result): “Similar to other crowdsourcing systems [Quinn and Bederson 2011; Yuen et al. 2011], broadly two types of incentives can be used in MCSC applications: intrinsic incentives or ﬁnancial incentives.”
Orienteering Problem: A survey of recent variants, solution approaches and applications (2016, background): “For a comprehensive survey about the crowdsourcing problem itself, we can refer to Yuen, King, and Leung (2011) .”