Extreme Clicking for Efficient Object Annotation

By Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, Vittorio Ferrari. 2017

259 citations recorded by the source corpus.

Manually annotating object bounding boxes is central to building computer vision datasets, and it is very time consuming (annotating ILSVRC [53] took 35s for one high-quality box [62]). It involves clicking on imaginary comers of a tight box around the object. This is difficult as these comers are often outside the actual object and several adjustments are required to obtain a tight box. We propose extreme clicking instead: we ask the annotator to click on four physical points on the object: the top, bottom, left- and right-most points. This task is more natural and these points are easy to find. We crowd-source extreme point annotations for PASCAL VOC 2007 and 2012 and show that (1) annotation time is only 7s per box, 5 × faster than the traditional way of drawing boxes [62]: (2) the quality of the boxes is as good as the original ground-truth drawn the traditional way: (3) detectors trained on our annotations are as accurate as those trained on the original ground-truth. Moreover, our extreme clicking strategy not only yields box coordinates, but also four accurate boundary points. We show (4) how to incorporate them into GrabCut to obtain more accurate segmentations than those delivered when initializing it from bounding boxes: (5) semantic segmentations models trained on these segmentations outperform those trained on segmentations derived from bounding boxes.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Human-Computer Interaction

Citations in the corpus

Segment Anything (2023, background): “5× faster than mask annotation for COCO [66] and only 2× slower than bounding-box labeling with extreme points [76, 71].”
Iteratively Trained Interactive Segmentation (2018, methodology): “In contrast to these methods, [20, 21] use extreme points on the objects to generate the corresponding segmentation masks.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “proposed a novel method for annotating objects by extreme clicks [25].”
nocaps: novel object captioning at scale (2019, neutral): “As with previous work, this task setting is also motivated by the observation that collecting human-annotated captions is resource intensive and scales poorly as object diversity grows, while on the other hand, large-scale object classification and detection datasets already exist [8,18] and their c”
The Open Images Dataset V4 (2018, methodology): “Afterwards, we co-invented extreme clicking (Papadopoulos et al., 2017).”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “ExtremeNet [79] groups ‘extreme points’ [57] according to the relation to a center point.”
Bottom-Up Object Detection by Grouping Extreme and Center Points (2019, methodology): “Although our training data is derived from the more expensive segmentation annotation, the extreme point data itself is 4× cheaper to collect than the standard bounding box [33].”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Bounding boxes take only 7 seconds to draw using the labelling method of [7], and image-level tags an average of 1 second per class [8].”
DOTA: A Large-Scale Dataset for Object Detection in Aerial Images (2017, methodology): “[23] have explored an alternative annotation method and verify its efficiency and robustness.”
Partially-Supervised Image Captioning (2018, background): “This reflects the observation that, in general, object detection datasets may be easier to scale — possibly semi-automatically [17, 18] — to new concepts than image caption datasets.”
Cross-Domain Weakly-Supervised Object Detection Through Progressive Domain Adaptation (2018, background): “Recently, [26] reduced it to 7 seconds through extreme clicking, while it still takes much time and effort to obtain large-scale datasets.”
Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection (2019, background): “Building datasets for a specific application can temporarily resolve this problem, nevertheless, the time and monetary costs incurred when manually annotating such datasets are not negligible [40, 33].”
Large-Scale Interactive Object Segmentation With Human Annotators (2019, methodology): “A well-tuned Grabcut implementation reaches 59% mIoU [26].”
One-shot Texture Segmentation (2018, methodology): “Segmentation Segmentation tasks can be roughly split into Semantic Segmentation [24, 6, 33] (including the recently introduced class-agnostic segmentation [25, 28]) and Instance Segmentation [30, 14].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, background): “2D keypoint annotations per image are straightforward to retrieve [25] and thus widely available [15,2,3].”
Deep Snake for Real-Time Instance Segmentation (2020, methodology): “Inspired by [32, 29, 45], we propose to generate an octagon formed by object extreme points as the initial contour, which generally encloses the object tightly.”
Interactive Object Segmentation With Inside-Outside Guidance (2020, background): “However, [49] argue that the annotated boxes are of low quality (not tight around the object).”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, background): “[46] propose extreme points and Maninis et al.”
Efficient Object Annotation via Speaking and Pointing (2019, methodology): “Recently,(Papadopoulos et al. 2017a) proposed extreme clicking, where a bounding box is annotated by clicking on four extreme points of the object.”