Recurrent Pixel Embedding for Instance Grouping

By Shu Kong, Charless C. Fowlkes. 2017

188 citations recorded by the source corpus.

We introduce a differentiable, end-to-end trainable framework for solving pixel-level grouping problems such as instance segmentation consisting of two novel components. First, we regress pixels into a hyper-spherical embedding space so that pixels from the same group have high cosine similarity while those from different groups have similarity below a specified margin. We analyze the choice of embedding dimension and margin, relating them to theoretical results on the problem of distributing points uniformly on the sphere. Second, to group instances, we utilize a variant of mean-shift clustering, implemented as a recurrent neural network parameterized by kernel bandwidth. This recurrent grouping module is differentiable, enjoys convergent dynamics and probabilistic interpretability. Backpropagating the group-weighted loss through this module allows learning to focus on correcting embedding errors that won't be resolved during subsequent clustering. Our framework, while conceptually simple and theoretically abundant, is also practically effective and computationally efficient. We demonstrate substantial improvements over state-of-the-art instance segmentation for object proposal generation, as well as demonstrating the benefits of grouping loss for classification tasks such as boundary detection and semantic segmentation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, methodology): “These methods work in a bottomup manner, generate embedding features [23], [24] for each pixel and use post-processing methods such as clustering [25], [26] or graph theory [27] to classify these pixels.”
On the Binding Problem in Artificial Neural Networks (2020, background): “to learn a similarity function between segments (Ren and Malik, 2003; Endres and Hoiem, 2010; Kong and Fowlkes, 2018) or discrete graphical patterns (Lun et al.”
Incremental Few-Shot Instance Segmentation (2021, neutral): “Approaches generally fall into two categories: grouping-based [4, 11, 17, 24] and proposal-based [2, 6, 15, 20] detection methods.”
AdaptIS: Adaptive Instance Selection Network (2019, background): “Another interesting approach to instance segmentation is based on instance embedding [5, 8, 16].”
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers (2021, methodology): “proaches [28, 1, 41, 44, 3, 29] regard segmentation as a bottom-up grouping task by first producing pixel-wise predictions followed by grouping object instances in the postprocessing stage.”
Self-Supervised Visual Representation Learning from Hierarchical Grouping (2020, unknown): “Kong et al. [27], adopting a similar objective, combine a CNN with a differentiable mean-shift clustering approach to learn instance segmentation.”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “For object centers that are out of view of the camera, we project them to the camera’s ﬁeld of view. c) Clustering Loss: We adopt a clustering loss that unrolls GMS for a few iterations and applies a loss on the clustered points, very similar to [72].”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, background): “The community has made great headway on this task recently [38, 39, 19, 17, 36, 10, 26, 2, 33, 21, 23].”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Other methods operate in a bottom-up fashion by grouping pixels belonging to the same object instance [18,19,43,59,60,61].”
Semi-convolutional Operators for Instance Segmentation (2018, methodology): “Some works have explored the use of pixellevel embeddings in the context of clustering tasks, employing them as a soft, differentiable proxy for cluster assignments [53,21,15,12,41,28].”
One-Shot Instance Segmentation (2018, methodology): “ semantic segmentation [49, 55, 73, 60, 90, 9, 15, 48], where every pixel is classiﬁed into a category, instance segmentation additionally requires to discriminate between individual object instances [27, 18, 28, 62, 19, 39, 63, 72, 5, 14, 23, 29, 45, 70, 37]. Most current state-of-the-art systems a”
Pixel-Wise Attentional Gating for Scene Parsing (2018, methodology): “…development of deep convolutional neural networks (CNN) has allowed remarkable progress in wide range of image pixel-labeling tasks such as boundary detection [41, 56, 29], semantic segmentation [31, 32, 6], monocular depth estimation [32, 36, 35, 38, 13], and surface normal estimation [53, 4, 12].”