Semi-convolutional Operators for Instance Segmentation

By David Novotný, Samuel Albanie, Diane Larlus, Andrea Vedaldi. 2018

88 citations recorded by the source corpus.

Object detection and instance segmentation are dominated by region-based methods such as Mask RCNN. However, there is a growing interest in reducing these problems to pixel labeling tasks, as the latter could be more efficient, could be integrated seamlessly in image-to-image network architectures as used in many other tasks, and could be more accurate for objects that are not well approximated by bounding boxes. In this paper we show theoretically and empirically that constructing dense pixel embeddings that can separate object instances cannot be easily achieved using convolutional operators. At the same time, we show that simple modifications, which we call semi-convolutional, have a much better chance of succeeding at this task. We use the latter to show a connection to Hough voting as well as to a variant of the bilateral kernel that is spatially steered by a convolutional network. We demonstrate that these operators can also be used to improve approaches such as Mask RCNN, demonstrating better segmentation of complex biological shapes and PASCAL VOC categories than achievable by Mask RCNN alone.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, methodology): “These methods work in a bottomup manner, generate embedding features [23], [24] for each pixel and use post-processing methods such as clustering [25], [26] or graph theory [27] to classify these pixels.”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, methodology): “Most of these methods [6, 18, 11, 5, 19] are based on an embedding loss function, which forces the feature vectors of pixels belonging to the same object to be similar to each other and sufficiently dissimilar from feature vectors of pixels belonging to other objects.”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “[25] showed that one necessarily needs to provide pixel coordinates to the network in order to disambiguate different object instances.”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, background): “The community has made great headway on this task recently [38, 39, 19, 17, 36, 10, 26, 2, 33, 21, 23].”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “[33] propose semi-convolutional operators based on the coordinates of each pixel for better instance segmentation.”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, methodology): “Although there are several attempts been made for using alternative approaches [7, 10, 27, 28, 29], Mask R-CNN still outperforms these methods in accuracy.”
Unseen Object Instance Segmentation for Robotic Environments (2020, background): “Recently, a few methods have investigated bottom-up methods which assign pixels to object instances [37], [38], [39], [40].”
Learning to Drive from Simulation without Real World Labels (2018, background): “The latent tensor Zd is passed through CoordConv [21] layers to learn a spatially aware embedding suited for control from this latent translation representation.”
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, background): “datasets, Convolutional Neural Networks (ConvNets) trained in a supervised setting are now state-of-the-art in many computer vision tasks such as object detection [10], instance segmentation [29], human pose estimation [28].”
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations (2019, methodology): “This idea is further explored in [21, 35], which predict instance centroids instead of box coordinates.”
Mind the Pad - CNNs can Develop Blind Spots (2020, background): “Whether or not this ability is desirable depends on the task, with several methods proposed to explicitly encode spatial information [4; 5; 18; 23; 28].”
Deep Hough Voting for 3D Object Detection in Point Clouds (2019, background): “[31] proposed semiconvolutional operator for 2D instance segmentation in images, which is also related to Hough voting.”
Explicit Shape Encoding for Real-Time Instance Segmentation (2019, methodology): “The mainstream methods can be roughly divided to top-down [12, 22, 3, 19, 28, 27, 2] or bottom-up [26, 35] approaches.”