Variational Inference for Crowdsourcing

By Qiang Liu, Jian Peng, A. Ihler. Neural Information Processing Systems, 2012

375 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/221eb399fff1ab8d5d260de0cd2809ca46652164

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Some Like it Hoax: Automated Fake News Detection in Social Networks (2017, background): “The BLC problem consists in computing the consensus labels from the user input [20, 21, 22].”
Lean Multiclass Crowdsourcing (2018, methodology): “Alternate inference algorithms for the DS model are based on spectral methods [7, 4, 12, 13, 14, 40], belief propagation [20, 23], expectation maximization [20, 40], maximum entropy [41, 42], weighted majority voting [19, 17] and max-margin [32].”
Error Rate Bounds in Crowdsourcing Models (2013, unknown): “M L ] 1 0 Ju l 2 01 3”
Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “…true label inference methods (Zheng et al., 2017), including majority voting (MV), ZenCrowd (Demartini et al., 2012), Mini-max (Zhou et al., 2012), LFC-binary (Raykar et al., 2010), CATD (Li et al., 2014a), PM-CRH (Aydin et al., 2014), KOS (Karger et al., 2011), VI-BP and VI-MF (Liu et al., 2012).”
Error Rate Analysis of Labeling by Crowdsourcing (2013, methodology): “, 2010) or using prior over worker confusion matrices (Raykar et al., 2010; Liu et al., 2012).”
Artificial Intelligence and Collective Intelligence (2014, methodology): “Liu et al. (Liu, Peng, and Ihler 2012) apply belief propagation and mean field approximation, techniques beyond the scope of this book, to perform the inference required to learn correct answers.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “Based on the above discussions, existing works [16, 15, 53, 51, 41, 33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] propose various ways to model a worker’s quality.”
Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
Bayesian representation learning with oracle constraints (2015, background): “Bayesian generative models have been proposed before for crowd-sourcing tasks (Liu et al., 2012).”
Crowdsourced Data Management: A Survey (2016, methodology): “Liu et al. [73] model workers and tasks using a directed graph, and apply variational inference methods to iteratively derive workers' models.”
Efficient Learning for Crowdsourced Regression (2017, methodology): “We propose a simple assignment scheme, referred to as (`, r)-regular task assignment, popularly adopted in crowdsourcing Dalvi et al. (2013); Liu et al. (2012); Karger et al. (2011, 2013, 2014); Ok et al. (2016).”
Aggregating crowdsourced binary ratings (2013, methodology): “Since then, there has been a host of followup work modifying this approach using a Bayesian technique [2, 11], studying it in the context of learning a specific classifier [14], and modifying it by finding out spammers, i.”
A Data Quality Metric (DQM): How to Estimate the Number of Undetected Errors in Data Sets (2016, methodology): “However, recent research suggests that this assumption holds true in practice [12, 22, 35]; Finally, our method can be combined with various prioritization strategies to determine the order of questions asked to the workers, and thus, is practical even for large data sets.”
Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing (2014, methodology): “The methods in [20, 19, 6] incorporate Bayesian inference into the Dawid-Skene estimator by assuming a prior over confusion matrices.”
Facial Expression Recognition with Inconsistently Annotated Datasets (2018, methodology): “The EM-based methods have had empirical success in determining the labels in crowdsourcing[3, 19, 36, 32].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, methodology): “[127] uses variational inference for worker quality management for filtering, and Zhou we al.”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, background): “The expected value of that posterior given a Beta (4 , 1) prior is (Liu et al., 2012): where i sums over the votes, and C i is one if the vote was correct and zero otherwise.”
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems (2011, unknown): “This observation was first suggested in Liu et al. (2012). Thus, our algorithm is belief propagation for a very specific prior.”
Learning From Noisy Singly-labeled Data (2017, methodology): “More sophisticated agreementbased algorithms jointly model worker skills and ground truth labels, estimating both using EM or similar techniques (Dawid & Skene, 1979; Jin & Ghahramani, 2003; Whitehill et al., 2009; Welinder et al., 2010; Zhou et al., 2012; Liu et al., 2012; Dalvi et al., 2013; Liu e”