Semantic Instance Segmentation via Deep Metric Learning

By Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, Kevin P. Murphy. 2017

204 citations recorded by the source corpus.

We propose a new method for semantic instance segmentation, by first computing how likely two pixels are to belong to the same object, and then by grouping similar pixels together. Our similarity metric is based on a deep, fully convolutional embedding model. Our grouping method is based on selecting all points that are sufficiently similar to a set of "seed points", chosen from a deep, fully convolutional scoring model. We show competitive results on the Pascal VOC instance segmentation benchmark.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, methodology): “These methods work in a bottomup manner, generate embedding features [23], [24] for each pixel and use post-processing methods such as clustering [25], [26] or graph theory [27] to classify these pixels.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
Mask Scoring R-CNN (2019, background): “Recently, there are some works [30, 11, 14, 10] us-ing metric learning to learn the embedding.”
YOLACT: Real-Time Instance Segmentation (2019, methodology): “Finally, some methods first perform semantic segmentation followed by boundary detection [22], pixel clustering [3, 25], or learn an embedding to form instance masks [32, 17, 9, 13].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “[38, 36] exploit boundary detection information, while [55, 23, 8] propose to cluster instances w.”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “5 for some recently proposed proposal detection methods, including unpublished work inst-DML [23] which is similar in spirit to our model but learns a Euclidean distance based metric to group pixels.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “Several methods [5, 17, 43] adopt instance level embeddings to differentiate instances.”
DeeperLab: Single-Shot Image Parser (2019, methodology): “On the other hand, the bottom-up approaches [73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84] generally adopt a two-stage processing: pixel-level predictions produced by the segmentation module are clustered together to form the instancelevel predictions.”
Affinity Derivation and Graph Merge for Instance Segmentation (2018, methodology): “The basic idea of these methods [34, 28, 15, 4] is to learn instance level features for each pixel with a CNN, then a clustering method is applied to group the pixels together.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “and semantic instance segmentation via deep metric learning [75].”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, methodology): “Most of these methods [6, 18, 11, 5, 19] are based on an embedding loss function, which forces the feature vectors of pixels belonging to the same object to be similar to each other and sufficiently dissimilar from feature vectors of pixels belonging to other objects.”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “based methods [8, 5, 25] which map the pixels of an image into embedding space and then perform clustering, AdaptIS does not require heuristic post-processing steps.”
Dense and Low-Rank Gaussian CRFs Using Deep Embeddings (2017, background): “We note that several other works have concurrently explored the use of embeddings in the context of grouping tasks, employing them as a soft, differentiable proxy for cluster assignments [11, 13, 14, 30].”
Semi-convolutional Operators for Instance Segmentation (2018, methodology): “It has been used for body joints [41], semantic segmentation [1,21,6] and optical flow [1], and, more relevant to our work, to instance segmentation [15,12,6,28].”
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, background): “Note that this loss is highly inﬂuenced by the instance segmentation in computer vision [21, 22].”
Learning About Objects by Learning to Interact with Them (2020, methodology): “Our model design is a convolutional neural network inspired by past works in clustering based instance segmentation [42, 16].”
One-Shot Instance Segmentation (2018, background): “In contrast to semantic segmentation [49, 55, 73, 60, 90, 9, 15, 48], where every pixel is classiﬁed into a category, instance segmentation additionally requires to discriminate between individual object instances [27, 18, 28, 62, 19, 39, 63, 72, 5, 14, 23, 29, 45, 70, 37].”
PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model (2018, methodology): “Similar to [2,61,62], we follow the embedding-based approach for this task.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “On the other hand, proposal-free methods [24, 7, 8, 20, 22] learn an embedding vector for each pixel and apply a clustering step in the embedding space as post processing for instance segmentation.”