Semantic Instance Segmentation with a Discriminative Loss Function

By Bert De Brabandere, Davy Neven, Luc Van Gool. 2017

479 citations recorded by the source corpus.

Semantic instance segmentation remains a challenging task. In this work we propose to tackle the problem with a discriminative loss function, operating at the pixel level, that encourages a convolutional network to produce a representation of the image that can easily be clustered into instances with a simple post-processing step. The loss function encourages the network to map each pixel to a point in feature space so that pixels belonging to the same instance lie close together while different instances are separated by a wide margin. Our approach of combining an off-the-shelf network with a principled loss function inspired by a metric learning objective is conceptually simple and distinct from recent efforts in instance segmentation. In contrast to previous works, our method does not rely on object proposals or recurrent mechanisms. A key contribution of our work is to demonstrate that such a simple setup without bells and whistles is effective and can perform on par with more complex methods. Moreover, we show that it does not suffer from some of the limitations of the popular detect-and-segment approaches. We achieve competitive performance on the Cityscapes and CVPPP leaf segmentation benchmarks.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “Although our accuracy does not meet state of the art methods [19, 12, 6], we find that for the car class category we are comparable to Color sequence:”
3D Instance Segmentation of MVS Buildings (2021, neutral): “Many other dense instance segmentation methods are based on metric learning [22].”
Incremental Few-Shot Instance Segmentation (2021, neutral): “Approaches generally fall into two categories: grouping-based [4, 11, 17, 24] and proposal-based [2, 6, 15, 20] detection methods.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
Mask Scoring R-CNN (2019, background): “Recently, there are some works [30, 11, 14, 10] us-ing metric learning to learn the embedding.”
YOLACT: Real-Time Instance Segmentation (2019, methodology): “Finally, some methods first perform semantic segmentation followed by boundary detection [22], pixel clustering [3, 25], or learn an embedding to form instance masks [32, 17, 9, 13].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “[38, 36] exploit boundary detection information, while [55, 23, 8] propose to cluster instances w.”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “One can utilize heuristic post-processing [18] or utilize clustering algorithms that estimate the number of instances [57], but these are not differentiable and thus unsatisfying.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “Several methods [5, 17, 43] adopt instance level embeddings to differentiate instances.”
DeeperLab: Single-Shot Image Parser (2019, methodology): “On the other hand, the bottom-up approaches [73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84] generally adopt a two-stage processing: pixel-level predictions produced by the segmentation module are clustered together to form the instancelevel predictions.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, background): “We observe, however, that there are other instance segmentation approaches based on initial semantic segmentation networks [16,28,29,30] which do not produce overlapping instances and can naturally handle “stuff” classes.”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, methodology): “Most of these methods [6, 18, 11, 5, 19] are based on an embedding loss function, which forces the feature vectors of pixels belonging to the same object to be similar to each other and sufficiently dissimilar from feature vectors of pixels belonging to other objects.”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “based methods [8, 5, 25] which map the pixels of an image into embedding space and then perform clustering, AdaptIS does not require heuristic post-processing steps.”
SOLO: Segmenting Objects by Locations (2019, background): “uping the pixels into an arbitrary number of object instances presented in an image. In [22], pixels are grouped into instances using the learned associative embedding. A discriminative loss function [7] learns pixel-level instance embedding eciently, by pushing away pixels belonging to dierent inst”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, unknown): “The grouping-based approaches [26, 2, 4, 35, 33, 1, 25] view the instance segmentation as a bottom-up grouping problem.”
Towards Bounding-Box Free Panoptic Segmentation (2020, methodology): “Towards this goal, a popular choice is to formulate it as an embedding problem using triplet loss [5].”
Unknown Object Segmentation from Stereo Images (2021, methodology): “Similar to [14], Xiang et al. [4] cluster pixel-wise feature representation predicted by a single convolutional network jointly trained on synthetic RGB and depth data.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Most IS approaches focus on how to represent instance masks beyond object detection, which can be divided into two categories: top-down approaches [76], [77] and bottom-up approaches [78], [79].”