Deep Interactive Object Selection

By Ning Xu, Brian L. Price, Scott D. Cohen, Jimei Yang, Thomas S. Huang. 2016

483 citations recorded by the source corpus.

Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep-learning-based algorithm which has much better understanding of objectness and can reduce user interactions to just a few clicks. Our algorithm transforms user-provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RGB channels of images to compose (image, user interactions) pairs. We generate many of such pairs by combining several random sampling strategies to model users' click patterns and use them to finetune deep Fully Convolutional Networks (FCNs). Finally the output probability maps of our FCN-8s model is integrated with graph cut optimization to refine the boundary segments. Our model is trained on the PASCAL segmentation dataset and evaluated on other datasets with different object classes. Experimental results on both seen and unseen objects demonstrate that our algorithm has a good generalization ability and is superior to all existing interactive object selection approaches.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “Accurate recognition and localisation of objects enables many applications, such as autonomous driving [8], image-editing [46] and robotics [16].”
Segment Anything (2023, methodology): “Our default point sampling follows standard practice in interactive segmentation [109, 64, 92].”
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “Thriving on large datasets, ConvNets [10, 30, 43, 47, 48] have evolved as the dominant architecture for high-quality interactive segmentation.”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “The basic paradigm [44] for click-based interactive segmentation is using Gaussian maps or disks to represent the clicks, then concatenating the click maps with the image as input, and sending it to a segmentation model to predict Coarse Segmentation Local Refinement User: Add Click”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “Click-based interactive segmentation is one of the most well-studied topics among other deep learning-based interactive segmentation approaches, as it has well-established protocols of training and evaluation [1, 2, 3, 4].”
Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “This is partly a consequence of existing benchmarks having set low accuracy targets for the task [6]: in the Berkeley benchmark[15] the threshold is set to 90% accuracy, 90% for the GrabCut dataset [16] and only 85% for PASCAL [5].”
Iteratively Trained Interactive Segmentation (2018, methodology): “[28], we train a convolutional network which takes an image and some user clicks as input and produces a segmentation mask (see Fig.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “Interactive User Input: The pipeline of DEXTR can also be used in the frame of interactive segmentation from points [41, 40].”
Guide Me: Interacting with Deep Networks (2018, methodology): “Nonetheless, some methods towards interactive deep learning have been proposed, such as weakly-supervised semantic segmentation from scribbles [38], user-provided clicks and Euclidean distance maps [61] or bounding boxes used as region initialization [17, 53].”
Real-time user-guided image colorization with learned deep priors (2017, methodology): “We also utilize design principles from (Simonyan and Zisserman 2014) and (Yu and Koltun 2016).”
Learning Video Object Segmentation from Static Images (2016, methodology): “Our approach outputs per-frame instance segmentation using a convnet architecture, inspired by works from other domains like [6, 44, 54].”
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation (2023, unknown): “The pioneering work of DIOS (Xu et al. 2016) Parameter-Efficient Fine-Turning PEFT has proven to be an efficient strategy for fine-tuning a large, fundamental model for a specific usage (Zaken, Rav-fogel, and Goldberg 2021).”
Interactive Image Segmentation with Latent Diversity (2018, methodology): “In particular, the state-of-the-art approach of Xu et al. [41], while based on convolutional networks, also relies heavily on postprocessing using graph cuts.”
Large-Scale Interactive Object Segmentation With Human Annotators (2019, background): “It has been revisited in the recent years under the deep learning umbrella [36, 20, 10, 34, 22, 16, 15, 1], focusing on how to best use clicks and scribbles corrections.”
Shifting More Attention to Video Salient Object Detection (2019, background): “…the underlying mechanism behind HVS during free-viewing in general and instrumental to a wide range of real-world applications, e.g ., video segmentation [74, 83], video captioning [57], video compression [27, 29], autonomous driving [91], robotic interaction [82], weakly supervised attention [95].”
Content-Aware Multi-Level Guidance for Interactive Instance Segmentation (2019, methodology): “The guidance map helps to select the speciﬁc instance to segment; when working in an iterative setting, it can also help correct errors from previous seg-mentations [6, 32, 35, 53].”
Interactive Video Object Segmentation in the Wild (2017, methodology): “For sampling negatives we use the same 3 strategies proposed by [18]: (i) randomly sampling in a hull of d hull = 40 around the object (ii) sampling negatives clicks from other objects in the image and (iii) sampling negatives as in (i), but such that they are maximally distant from each other, i.e…”
Simple Does It: Weakly Supervised Instance and Semantic Segmentation (2016, methodology): “Inspired by [45, 4], we modify DeepLab to accept four input channels: the input image RGB channels, plus a binary map with a bounding box of the object instance to segment.”
PseudoClick: Interactive Image Segmentation with Click Imitation (2022, methodology): “While different interaction types have been investigated, including clicks [10,11], bounding boxes [12,13,14], and scribbles [15,16], we only focus on click-based interactive segmentation because it has the simplest interaction and well-established ar X iv :2 20 7.”