Interactive Image Segmentation with Latent Diversity

By Zhuwen Li, Qifeng Chen, Vladlen Koltun. 2018

221 citations recorded by the source corpus.

Interactive image segmentation is characterized by multimodality. When the user clicks on a door, do they intend to select the door or the whole house? We present an end-to-end learning approach to interactive image segmentation that tackles this ambiguity. Our architecture couples two convolutional networks. The first is trained to synthesize a diverse set of plausible segmentations that conform to the user's input. The second is trained to select among these. By selecting a single solution, our approach retains compatibility with existing interactive segmentation interfaces. By synthesizing multiple diverse solutions before selecting one, the architecture is given the representational power to explore the multimodal solution space. We show that the proposed approach outperforms existing methods for interactive image segmentation, including prior work that applied convolutional networks to this problem, while being much faster.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Segment Anything (2023, neutral): “During training, we backprop only the minimum loss [15, 45, 64] over masks.”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “DAVIS is firstly introduced into interactive segmentation in [22].”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “To generate clicks during the evaluation procedure, we follow the strategy described in [7].”
Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “In IIS-LD [11], a single full resolution network is proposed by using VGG hypercolumns.”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “On the 2D image side, we use a VGG-16 like architecture to extract features as it has been demonstrated to be successful for many tasks [10, 20].”
Deep Interactive Thin Object Selection (2021, methodology): “Therefore, at below we ﬁrst introduce a new large-scale dataset for thin object segmen-1 , Latent Diversity [21], f-BRS [41] and DEXTR [32].”
MultiSeg: Semantically Meaningful, Scale-Diverse Segmentations From Minimal User Input (2019, methodology): “Following [23, 36], we apply graph cut [4] to the network prediction to obtain the final segmentation mask.”
FocusCut: Diving into a Focus View in Interactive Segmentation (2022, methodology): “[26] couple two convolutional networks to train and select the proper result.”
Content-Aware Multi-Level Guidance for Interactive Instance Segmentation (2019, methodology): “It should be noted that FCTSFN [26] and IIS-LD [31] report their result over all the 80 classes of MS COCO and not separately for 20 seen and 60 unseen classes.”
A Fully Convolutional Two-Stream Fusion Network for Interactive Image Segmentation (2018, methodology): “Li et al. [28] use a segmentation network to generate various potential segmentations from image and user interactions; then a selection network is applied to select the output from the potential segmentations.”
Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search (2018, methodology): “To train f to generate diverse high-quality probability maps, we adopt the hindsight loss [18, 8, 28]:”
Deepstrip: High-Resolution Boundary Refinement (2020, background): “Recent decades have witnessed considerable effort to improve the boundary quality of an object that has been detected [41, 35, 32, 18, 37, 47, 16, 22] or segmented [11, 34, 24, 7].”
Interactive Full Image Segmentation by Considering All Regions Jointly (2018, methodology): “Our approach of full image segmentation brings several advantages over modern interactive single object segmentation methods [24, 30, 31, 32, 36, 37, 60]: (I) It enables the annotator to focus on the largest errors in the whole image, rather than on the largest error of one given object.”
F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (2020, methodology): “On the other hand, deep learning-based methods [31, 18, 22] tend to overuse image semantics.”
Conditional Diffusion for Interactive Segmentation (2021, background): “[17] predicts multiple potential results and train another network to choose from them.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “i ;) over the training set using stochastic gradient descent. 3.2. Learning from Corrections at Test-Time In previous interactive object segmentation works all model parameters are ﬁxed at test time [52,10,35,31,11, 28]. Corrections are only used as inputs to guide the predictions. Instead, we treat”