Iterative Learning for Reliable Crowdsourcing Systems

By Bin Bi, Chen Liu, Yuchen Liu. 2012

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, methodology): “[121] passively collects data and runs the expectation maximization algorithm.”
Data Programming: Creating Large Training Sets, Quickly (2016, background): “More recent results provide formal guarantees even in the absence of labeled data using various approaches [3,8,14,15,22,31].”
Counting with the Crowd (2012, background): “The approach to getting an item’s label (e.g., identifying a picture of a ﬂower as being blue ) through multiple crowd workers is well-studied [14, 15].”
Efficient Learning for Crowdsourced Regression (2017, methodology): “We propose a simple assignment scheme, referred to as (`, r)-regular task assignment, popularly adopted in crowdsourcing Dalvi et al. (2013); Liu et al. (2012); Karger et al. (2011, 2013, 2014); Ok et al. (2016).”
SLiMFast: Guaranteed Results for Data Fusion and Source Reliability (2015, background): “The only work prior to ours that comes with theoretical guarantees on the error rate of accuracy estimates is from the crowdsourcing community [8, 17, 20, 18] where data sources correspond to human workers.”
Some Like it Hoax: Automated Fake News Detection in Social Networks (2017, background): “Furthermore, while the harmonic algorithm can be adapted to the presence of a learning set, it is less obvious how to do so for some of the other algorithms, such as those of [13].”
Pairwise ranking aggregation in a crowdsourced setting (2013, methodology): “(11), (12), (13), (14), (16) and (17). end for Output: Rank objects by sorting the obtained {μi}.”
Query-Oriented Data Cleaning with Oracles (2015, methodology): “In addition, previous works propose different methods for evaluating crowd workers’ quality, e.g., to ﬁlter spam-mers and identify domain experts [42, 39, 36, 29, 37].”
Lean Multiclass Crowdsourcing (2018, methodology): “Alternate inference algorithms for the DS model are based on spectral methods [7, 4, 12, 13, 14, 40], belief propagation [20, 23], expectation maximization [20, 40], maximum entropy [41, 42], weighted majority voting [19, 17] and max-margin [32].”
Decision Tree Design for Classification in Crowdsourcing Systems (2018, background): “These problems have also been extended to budget-constrained environments to improve classification performance [10]–[12].”
Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “…true label inference methods (Zheng et al., 2017), including majority voting (MV), ZenCrowd (Demartini et al., 2012), Mini-max (Zhou et al., 2012), LFC-binary (Raykar et al., 2010), CATD (Li et al., 2014a), PM-CRH (Aydin et al., 2014), KOS (Karger et al., 2011), VI-BP and VI-MF (Liu et al., 2012).”
A brief introduction to weakly supervised learning (2018, background): “For this purpose, non-adaptive task assignment mechanisms which assign tasks off-line [47, 74], and adaptive mechanisms which assign tasks online [18, 41], have both been studied with theoretical supports.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “Based on the above discussions, existing works [16, 15, 53, 51, 41, 33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] propose various ways to model a worker’s quality.”
Crowdsourcing large scale wrapper inference (2014, background): “, [23]) in which redundancy is established statically, i.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, background): “1 Introduction There is an increasing interest in using crowdsourcing to collect labels for machine learning [19, 6, 21, 17, 20, 10, 13, 12].”