The Multidimensional Wisdom of Crowds

By Peter Welinder, Steve Branson, Serge J. Belongie, Pietro Perona. 2010

908 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

ImageNet Large Scale Visual Recognition Challenge (2014, background): “Works such as (Welinder et al., 2010; Sheng et al., 2008; Vittayakorn and Hays, 2011) describe quality control mechanisms for this marketplace.”
Extreme Clicking for Efficient Object Annotation (2017, background): “Quality control is a common process when crowd-sourcing image annotations [4, 36, 43, 53, 56, 60, 62, 66, 70].”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “The crowdsourcing literature has studied how to increase accuracy of noisy labelers via rater modeling [11, 56] or clever construction of easier tasks based on attributes, parts, or visual similarity [55, 6, 36].”
Towards a Visual Turing Challenge (2014, background): “Crowdsourci ng in combination of machine learning approaches have served us well to generate curated datasets with a unique ground truth at scale [33, 34].”
Discovering Attribute Shades of Meaning with the Crowd (2015, background): “Aggregating crowd labels Crowd input has been aggregated in novel ways for image clustering (Gomes et al, 2011), image similarity (Tamuz et al, 2011), and object labeling (Welinder et al, 2010).”
Perceptual Annotation: Measuring Human Vision to Improve Computer Vision (2014, background): “, eye movements used to define discriminative image regions for feature extraction [14], [15], structured domain knowledge from human experts [16], [17], [18], and models of the typical human annotation process itself [19].”
Boundary Detection Benchmarking: Beyond F-Measures (2013, background): “Many seminal models [18, 26] have been proposed to analyze the crowdsourced annotation process in general.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, background): “1 Introduction There is an increasing interest in using crowdsourcing to collect labels for machine learning [19, 6, 21, 17, 20, 10, 13, 12].”
Counting in the Wild (2016, neutral): “Dealing with multiple annotators has been generally approached by modelling different annotation variables with the objective of scoring and weighting the influence of each of the annotators [12, 22, 23], and finding the ground-truth label that is assumed to exist in the consensus of the annotators ”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, background): “Quality control is a common process when crowd-sourcing image annotations [4, 36, 43, 53, 56, 60, 62, 66, 70].”
Learning Whom to Trust with MACE (2013, methodology): “In machine vision, several models have taken this further by parameter-izing difﬁculty in terms of complex features deﬁned on each pairing of annotator and annotation instance (Welinder et al., 2010; Yan et al., 2010).”
Error Rate Bounds in Crowdsourcing Models (2013, result): “However, if we can appropriately aggregate the outputs from a crowd, the aggregated results could be as good as the results by an expert [1, 5, 6, 8, 9, 10, 11, 12, 13].”
SURF: improving classifiers in production by learning from busy and noisy end users (2020, background): “[11] provide another Bayesian treatment of the problem, again considering worker competence rather than a confusion matrix approach.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “Based on the above discussions, existing works [16, 15, 53, 51, 41, 33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] propose various ways to model a worker’s quality.”
Aggregating crowdsourced binary ratings (2013, background): “Other related results in applying machine learning techniques to cleaning user labels include [19, 16, 14, 18].”
Teaching Categories to Human Learners with Visual Explanations (2018, methodology): “[36], we propose to use the features from our finetuned CNN from the previous section.”
Bootstrapping Fine-Grained Classifiers : Active Learning with a Crowd in the Loop (2013, background): “Obtaining high quality results from crowdsourced annotators, who are often untrained and of unknown provenance, is open research topic [21, 10, 12].”
Crowdsourcing in Computer Vision (2016, neutral): “Yao et al. [2012] present an iterative framework consisting of four steps: (1) object hypotheses are generated via a Hough Forest-based object detector, (2) the hypotheses are corrected by the user, (3) the detector is incrementally updated using the new labels, and (4) new hypotheses are generated ”
Learning visual biases from human imagination (2014, background): “nd into object recognition is inspired by many recent works that puts a human in the computer vision loop [5,10,29], trains recognition systems with active learning [36,34], and studies crowdsourcing [37,32,40]. The primary difference of these approaches and our work is, rather than using crowds as ”