Crowdsourcing Annotations for Visual Object Detection

By Hao Su, Jia Deng, Li Fei-Fei. 2012

309 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ImageNet Large Scale Visual Recognition Challenge (2014, methodology): “We summarize the crowdsourced bounding box annotation system described in detail in (Su et al., 2012).”
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation (2013, background): “More information on how ILSVRC was collected and annotated can be found in [11, 36].”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, background): “5 seconds required for drawing a bounding box around an object [35].”
Extreme Clicking for Efficient Object Annotation (2017, methodology): “Quality control is a common process when crowd-sourcing image annotations [4, 36, 43, 53, 56, 60, 62, 66, 70].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “ey are likely to result in mentions being associated with incorrect entities or image regions. 3.2 Bounding Box Annotations The workﬂow to collect bounding box annotations is broken down similarly to Su et al. (2012), and consists of four separate AMT tasks, discussed below: (1) Box Requirement, (2)”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “Due to legal or technical constraints, and the fact that labeling data is expensive and time-consuming [34], real-world deep-learning pipelines would rarely involve a single fine-tuning stage.”
What's the Point: Semantic Segmentation with Point Supervision (2015, methodology): “The protocol of [36] has been used for producing the official annotations of the ILSVRC [31], which is currently the most popular dataset for object class detection and is of comparable difficulty to PASCAL VOC.”
Bottom-Up Object Detection by Grouping Extreme and Center Points (2019, neutral): “5 seconds on average [44].”
GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild (2018, methodology): “We follow the standard in object detection [25] for the labeling”
Interactive Object Segmentation With Inside-Outside Guidance (2022, background): “However, it was previously reported that drawing a tight box can be time consuming ([84] reported 25.”
Re-id: Hunting Attributes in the Wild (2014, background): “However, crowd-sourced annotation at scale still incurs expenses in terms of time and human effort, and the results are often prone to bias and noise [32].”
Omni-DETR: Omni-Supervised Object Detection with Transformers (2022, unknown): “The annotation cost, in seconds per image, for each type of annotation is shown in Table 11, following [2, 31, 35, 43].”
3D Object Representations for Fine-Grained Categorization (2013, methodology): “After obtaining images for each of the 197 target classes, we collect a bounding box for each image via AMT, using a quality-controlled system provided to us by the authors of [28].”
Region-Based Convolutional Networks for Accurate Object Detection and Segmentation (2016, background): “collected and annotated can be found in [65], [66].”
Efficient Learning for Crowdsourced Regression (2017, background): “…on crowdsourcing platforms require continuous valued evaluation of a training dataset, e.g., the location of an object Everingham et al. (2015); Su et al. (2012), center of a galaxy, or the center of a marker in a Cryo-EM image, we lack systematic study of how to tackle the human noise in thus…”
Embracing Error to Enable Rapid Crowdsourcing (2016, methodology): “, and information gain can reduce the number of labels necessary to build large taxonomies using a crowd [11, 4]. Methods have also been proposed to maximize accuracy of object localization in images [61] and videos [66]. Previous labels can also be used as a prior to optimize acquisition of new typ”
Efficient Object Annotation via Speaking and Pointing (2019, methodology): “Bounding box annotation Typically, bounding boxes are annotated given image-level labels (Russakovsky et al. 2015a; Kuznetsova et al. 2018; Lin et al. 2014; Su et al. 2012; Russakovsky et al. 2015a).”
Best of both worlds: Human-machine collaboration for object annotation (2015, methodology): “[50] describes Step 2 and reports time per bounding box (including quality control) as 42.”
Image retrieval using scene graphs (2015, methodology): “Bounding boxes for objects are corrected and veri-ﬁed using a system similar to [60], and all tuples are veriﬁed by other workers.”