Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

By Davy Neven, Bert De Brabandere, Marc Proesmans, Luc Van Gool. 2019

256 citations recorded by the source corpus.

Current state-of-the-art instance segmentation methods are not suited for real-time applications like autonomous driving, which require fast execution times at high accuracy. Although the currently dominant proposal-based methods have high accuracy, they are slow and generate masks at a fixed and low resolution. Proposal-free methods, by contrast, can generate masks at high resolution and are often faster, but fail to reach the same accuracy as the proposal-based methods. In this work we propose a new clustering loss function for proposal-free instance segmentation. The loss function pulls the spatial embeddings of pixels belonging to the same instance together and jointly learns an instance-specific clustering bandwidth, maximizing the intersection-over-union of the resulting instance mask. When combined with a fast architecture, the network can perform instance segmentation in real-time while maintaining a high accuracy. We evaluate our method on the challenging Cityscapes benchmark and achieve top results (5% improvement over Mask R-CNN) at more than 10 fps on 2MP images.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Recently, PanopticDeepLab [20], a simple, fast, and strong approach for bottom-up panoptic segmentation, employs a class-agnostic instance segmentation branch involving a simple instance center regression [45,82,66], coupled with DeepLab semantic segmentation outputs [13,15,16].”
K-Net: Towards Unified Image Segmentation (2021, methodology): “‘Bottom-up’ methods [1, 30, 43, 44] first perform semantic segmentation then group pixels into different instances.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, instance segments are obtained by grouping ‘thing’ pixels with various methods, such as instance center regression [44, 86, 70, 100, 20], Watershed transform [88, 3, 8], Hough-voting [4, 53, 8], or pixel affinity [45, 66, 81, 30, 8].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “Several methods [5, 17, 43] adopt instance level embeddings to differentiate instances.”
k-means Mask Transformer (2022, methodology): “…clustering-based methods usually operate on semantic segments [13,15,18] and group ‘thing’ pixels into instance segments with various representations, such as instance center regression [50,89,77,104,22,94,63], Watershed transform [91,5], Hough-voting [6,59,92], or pixel aﬃnity [51,69,85,36,47].”
BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation (2020, background): “Recognizing these difﬁculties, a line of research takes a bottom-up strategy [1, 19, 20].”
Cellpose: a generalist algorithm for cellular segmentation (2020, background): “These ﬂows do not necessarily point directly towards the center of the cell [51], because that direction might intersect cell boundaries.”
Deep Snake for Real-Time Instance Segmentation (2020, methodology): “[31] differentiates instances by learning instance-level embeddings.”
Unknown Object Segmentation from Stereo Images (2021, methodology): “Even anchor-free methods [9], [10] then require expensive post-processing steps such as Non-Maximum Suppression [11], watershed algorithm [12], hough-voting [13], or clustering [14], [15] to filter the raw network outputs.”
RTMDet: An Empirical Study of Designing Real-Time Object Detectors (2022, methodology): “Pioneer methods explore different paradigms to tackle this task, including mask classification [60, 61], ‘Top-Down’ [8, 31], and ‘Bottom-Up’ approaches [1, 39, 58].”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, methodology): “The bottom-up methods [9, 36] widely employ multiple stages to first perform semantic segmentation and then identify the specific location of each instance by boundary detection [20], pixel clustering [25], or pixels embedding loss [30, 22, 31], position-sensitive pooling [11, 24].”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, background): “[30] introduce a learnable clustering bandwidth instead of learning embedding using hand-crafted cost functions, achieving accurate instance segmentation in real-time.”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “With a broader view, a few embedding based instance segmentation approaches [26, 86] can be viewed as nonparametric, i.”
Segment as Points for Efficient Online Multi-Object Tracking and Segmentation (2020, methodology): “For APOLLO MOTS, we train SpatialEmbedding from scratch following [24].”
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location (2020, background): “Explicitly adding absolute spatial location information helps for patch matching [67, 71], generative modeling [101], semantic segmentation [36, 100], instance segmentation [72].”
YOLACT++ Better Real-Time Instance Segmentation (2019, methodology): “While [40] substantially improves instance segmentation accuracy over these prior methods, it runs only at 11 fps on Cityscapes.”
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, methodology): “Existing instance segmentation methods either follow bottom-up [1,26,30,33,19] or top-down [21,31,8,2,36] paradigms.”