Crowdsourcing in Computer Vision

By Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman. 2016

168 citations recorded by the source corpus.

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. Crowdsourcing in Computer Vision describes the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. It begins by discussing data collection on both classic vision tasks, such as object recognition, and recent vision tasks, such as visual story-telling. It then summarizes key design decisions for creating effective data collection interfaces and workflows, and presents strategies for intelligently selecting the most important data instances to annotate. It concludes with some thoughts on the future of crowdsourcing in computer vision. Crowdsourcing in Computer Vision provides an overview of how crowdsourcing has been used in computer vision, enabling a computer vision researcher who has previously not collected non-expert data to devise a data collection strategy. It will also be of help to researchers who focus broadly on crowdsourcing to examine how the latter has been applied in computer vision, and to improve the methods that can be employed to ensure the quality and expedience of data collection.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

Citations in the corpus

Global-and-local attention networks for visual recognition (2018, background): “Previous work has shown that it is possible to augment vision systems with human perceptual judgments on difﬁcult problems including face recognition and localization [23–25], bird categorization [26], action recognition [27], and object detection and segmentation [28–31].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “This, in turn, would require a tremendous amount of manual annotations, despite the expert knowledge in crowdsourcing accumulated by the community over the years [30].”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “work flows of crowdsourcing object annotations [79], [80], [81], all our 2D/3D labeling tasks, e.”
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk (2017, background): “that is widely used by industry [32,46] and the HCI community, as well as by other research areas such as NLP and computer vision [15,43].”
Human Uncertainty Makes Classification More Robust (2019, background): “Much of crowdsourcing work focuses on reconciling human labels and mitigating their disagreement (c.f., Ko-vashka et al. [25] for a survey).”
A survey of public datasets for computer vision tasks in precision agriculture (2020, unknown): “This dataset has been widely used for developing robotic crop and weed detection algorithms (Bosilj et al., 2020a; Buhrmester et al., 2011b; Kovashka et al., 2016; Lottes et al., 2018).”
Lean Multiclass Crowdsourcing (2018, background): “[15] provide a thorough review of crowdsourcing techniques for computer vision.”
What are the Visual Features Underlying Human Versus Machine Vision? (2017, background): “Researchers have found that in cases like these human perceptual judgments can augment the performance of vision models [35], resulting in significant gains in face recognition and localization [24, 2, 14], action recognition [31], object detection and segmentation [20, 26, 32].”
Unsupervised Single-Scene Semantic Segmentation for Earth Observation (2022, background): “Most computer vision problems can use crowdsourcing [12] to collect large volume of labeled data.”