Large-Scale Interactive Object Segmentation With Human Annotators

By Rodrigo Benenson, Stefan Popov, Vittorio Ferrari. 2019

260 citations recorded by the source corpus.

Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We released this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3x faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “This is in line with the 90 IoU human agreement per instance reported for OpenImages [5] and much higher than the ≈80 IoU agreement reported for MS-COCO polygons in the OpenImages paper [5].”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, background): “Annotating data with segmentation masks is very time-consuming and therefore expensive, but interactive segmentation can significantly simplify and speed up this process [3].”
Connecting Vision and Language with Localized Narratives (2019, methodology): “ages of COCO (train and validation). For Open Images, we annotated 377;591 images, selected from the train split. To enable cross-modal applications, we selected images for which object segmentations [5], object bounding boxes or visual relationship annotations [31] are already available. All the an”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “OPENIMAGES V5 [61], [62], [63] Everyday Objects 957,561 Provides only object segmentations (no segmentation of stuff categories).”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2022, background): “OPENIMAGES V5 [61], [62], [63] Everyday Objects 957,561 Provides only object segmentations (no segmentation of stuff categories).”
Interactive Object Segmentation With Inside-Outside Guidance (2022, methodology): “According to the human inputs, the interactive segmentation approached can be roughly divided into click-based [1], [37], [38], [47], [48], [49], [50], [51] and polygon-based [52], [53], [54].”
iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images (2021, methodology): “We encode positive and negative clicks from coordinates to a 2-channel feature map with the same spatial size with the input image, following the strategy used in [25].”
Object Detection in 20 Years: A Survey (2019, unknown): “, ILSVRC2014) [56], the MS-COCO Detection Challenge [57], the Open Images Dataset [58], [59], Objects365 [60], and so on.”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “We use the instance masks from OID [82] and show results in Table IV.”
PseudoClick: Interactive Image Segmentation with Click Imitation (2022, methodology): “We first encode clicks from coordinates to spatial form using a disk map [31], which encodes each click as a disk representation with a fixed radius r.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “e not known. In addition, some previous works [31] predict at the full input image resolution, whereas our backbone network predicts at 1/4 of it. If we ignore objects smaller than 80 80 pixels as in [11], our IA+SA method improves from9.9to5.6 clicks@85% on the unseen classes of COCO. Importantly, ”
F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (2020, methodology): “Annotation of these datasets became feasible with the use of automated interactive segmentation methods [1, 3].”
EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow (2021, methodology): “Inspired by [6, 47], in our implementation, we utilize the disk map with a radius of 5 pixels to encode both positive and negative clicks.”
An Empirical Study of Detection-Based Video Instance Segmentation (2019, methodology): “In order to increase data diversity, we expand our training data to include a subset of COCO [16] and OpenImage [2] data sets.”
Pointly-Supervised Instance Segmentation (2021, methodology): “For instance segmentation, point clicks are often used in interactive pipelines [5, 29, 30, 35, 58].”
Deepstrip: High-Resolution Boundary Refinement (2020, background): “Recent decades have witnessed considerable effort to improve the boundary quality of an object that has been detected [41, 35, 32, 18, 37, 47, 16, 22] or segmented [11, 34, 24, 7].”
FocusCut: Diving into a Focus View in Interactive Segmentation (2022, background): “The combination of interactions, such as the bounding box and clicks [4, 52], is also popular in the field.”