Best of both worlds: Human-machine collaboration for object annotation

By Olga Russakovsky, Li-Jia Li, Li Fei-Fei. 2015

219 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop (2015, methodology): “Clever methods have been proposed to improve the efﬁciency of human-in-the-loop annotation (e.g., [2, 5, 13, 20, 21]).”
Extreme Clicking for Efficient Object Annotation (2017, background): “We also compare to two methods which strike a trade-off between accuracy and efficiency [54, 47].”
YOLOv3: An Incremental Improvement (2018, background): “[18] If humans have a hard time telling the difference, how much does it matter?”
What's the Point: Semantic Segmentation with Point Supervision (2015, background): “For comparison, [32] reports a higher 25% average error rate when drawing bounding boxes.”
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images (2016, background): “[17] propose a framework that combines object detectors with human annotators to annotate a dataset.”
A scalable active framework for region annotation in 3D shape collections (2016, methodology): “Request”
Shallow and Deep Convolutional Networks for Saliency Prediction (2016, background): “The rise of convnets originated with image classification [23] in the context of increasing availability of annotated data.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, unknown): “Supervised object detection requires costly annotation efforts – annotating the bounding box of a single object takes up to 42 seconds [195].”
Efficient Object Annotation via Speaking and Pointing (2019, methodology): “Bounding box annotation Typically, bounding boxes are annotated given image-level labels (Russakovsky et al. 2015a; Kuznetsova et al. 2018; Lin et al. 2014; Su et al. 2012; Russakovsky et al. 2015a).”
Embracing Error to Enable Rapid Crowdsourcing (2016, methodology): “ Current methods calculate the minimum redundancy necessary to be conﬁdent that errors have been removed [54 ,57 58]. These methods typically result in a 0.25 to 1 speedup beyond a ﬁxed majority vote [45,50,54,27]. We take the opposite position: that designing the task to encourage some error, or ev”
Interactive Object Segmentation With Inside-Outside Guidance (2020, background): “[54] reported 10.21 s while [19] reported 7.0 s ).”
Mining Object Parts from CNNs via Active Question-Answering (2017, background): “Inspired by active learning [38, 41, 22], the idea of learning from questionanswering has been used to learn object models [9, 27, 39].”
Crowdsourcing in Computer Vision (2016, unknown): “In contrast, the scale of ImageNet Large Scale Visual Recognition Challenge (ILSVRC) object detection [Russakovsky et al., 2015a] makes it impossible to annotate in-house.”
Interactive Full Image Segmentation by Considering All Regions Jointly (2018, background): “The avant-garde work of [49] had a machine dispatching many labeling questions to annotators, including whether an object class is present, box verification, box drawing, and finding missing instances of a particular class in the image.”
Consistency-based Semi-supervised Learning for Object detection (2019, background): “Labeling for object detection requires a pair of a category and a bounding box location for each object within each image and it is known that it takes about 10 seconds for labeling an object [1, 2].”
Learning to Track: Online Multi-object Tracking by Decision Making (2015, background): “Markov decision processes [6] have been applied to different computer vision tasks, such as feature selection for recognition [35, 17], human activity forecasting [21], video game playing [28] and human-machine collaboration [37].”
Interactive Object Segmentation With Inside-Outside Guidance (2022, background): “Some papers reported much faster timings (e.g. [85] reported 10.21 s while [86] reported 7.0 s ).”
I Have Seen Enough: Transferring Parts Across Categories (2016, background): “The work of [56] estimates a cost of different types of annotations and then modulates an expected risk function while the strategy of [40] annotates as many examples in each image as possible.”
We Don’t Need No Bounding-Boxes: Training Object Class Detectors Using Only Human Verification (2016, background): “[41] propose such a scheme to fully detect all objects in images of complex scenes.”