Learning from the Wisdom of Crowds by Minimax Entropy

By Dengyong Zhou, John C. Platt, Sumit Basu, Yi Mao. 2012

394 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Why Amazon's Ratings Might Mislead You: The Story of Herding Effects (2014, neutral): “Indeed, models on collective intelligence, from majority voting to collaborating ﬁltering 9–12 and crowdsourcing, 13,14 all assume independent crowds, representing a critical gap between modeling frameworks and empirical insights.”
Quantifying herding effects in crowd wisdom (2014, background): “While the crowd wisdom applies usefully to a spectrum of domains, ranging from product or service recommendation [10] and crowdsourcing [5, 20, 26] to stock markets and political elections [22], one key prerequisite of harnessing the crowd wisdom is the independency of individuals’ opinions [22].”
Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning (2014, background): “entropy-based AL can be eective in many situations [36, 40], in many other situations, when the classiers do not produce accurate probabilities, entropy does not guarantee an unbiased estimate of”
Lean Multiclass Crowdsourcing (2018, methodology): “Alternate inference algorithms for the DS model are based on spectral methods [7, 4, 12, 13, 14, 40], belief propagation [20, 23], expectation maximization [20, 40], maximum entropy [41, 42], weighted majority voting [19, 17] and max-margin [32].”
Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing (2014, background): “Multi-class tasks include labeling the breed of dogs from ImageNet [9] (Dog dataset) and judging the relevance of web search results [26] (Web dataset).”
Error Rate Analysis of Labeling by Crowdsourcing (2013, background): “In spite of these drawbacks, is it still possible to get reliable answers in a crowdsourcing system? In fact, majority voting has been able to generate fairly reasonable results (Snow et al., 2008; Raykar et al., 2010; Liu et al., 2012; Zhou et al., 2012).”
Error Rate Bounds in Crowdsourcing Models (2013, unknown): “M L ] 1 0 Ju l 2 01 3”
A Survey on Truth Discovery (2015, background): “Truth discovery is motivated by the strong need to resolve conﬂicts among multi-source noisy information, since con-ﬂicts are commonly observed in database [8,9,17], the Web [29, 52], crowdsourced data [41, 42, 59], etc.”
Embracing Error to Enable Rapid Crowdsourcing (2016, methodology): “Finally, a generative model can be used to model workers’ skills by minimizing the entropy of the distribution over their labels and the unknown true labels [73].”
Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “We compare our proposed framework GroupAnno with nine existing true label inference methods (Zheng et al., 2017), including majority voting (MV), ZenCrowd (Demartini et al., 2012), Mini-max (Zhou et al., 2012), LFC-binary (Raykar et al., 2010), CATD (Li et al., 2014a), PM-CRH (Aydin et al., 2014), K”
Scalable multi-label annotation (2014, background): “While the problem of acquiring one label has been well studied [20, 9, 22, 19, 16, 5], to our knowledge the challenge of largescale multi-label annotation has not been addressed before.”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, unknown): “[202, 203] use minimax entropy.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “The single-choice tasks are especially studied in [34, 16, 15, 53, 41, 61, 35, 30, 27, 46, 5].”
Learning From Noisy Singly-labeled Data (2017, methodology): “More sophisticated agreementbased algorithms jointly model worker skills and ground truth labels, estimating both using EM or similar techniques (Dawid & Skene, 1979; Jin & Ghahramani, 2003; Whitehill et al., 2009; Welinder et al., 2010; Zhou et al., 2012; Liu et al., 2012; Dalvi et al., 2013; Liu e”
Efficient Learning for Crowdsourced Regression (2017, background): “…a variety of real-world tasks ranging from classification Sheng et al. (2008) and regression Marcus et al. (2012) to more complex tasks such as protein folding Peng et al. (2013), searching videos Bernstein et al. (2011), ranking Lee et al. (2012) and natural language processing Wu et al. (2012).”
A Confidence-Aware Approach for Truth Discovery on Long-Tail Data (2014, background): “Another related field is learning from crowd of wisdom, also known as crowdsourcing, in which researchers investigate how to infer true labels from the labeling efforts of a crowd [3, 15, 29, 30, 32, 33, 38].”