Recurrent Instance Segmentation

By Bernardino Romera-Paredes, Philip H. S. Torr. 2015

334 citations recorded by the source corpus.

Instance segmentation is the problem of detecting and delineating each distinct object of interest appearing in an image. Current instance segmentation approaches consist of ensembles of modules that are trained independently of each other, thus missing opportunities for joint learning. Here we propose a new instance segmentation paradigm consisting in an end-to-end method that learns how to segment instances sequentially. The model is based on a recurrent neural network that sequentially finds objects and their segmentations one at a time. This net is provided with a spatial memory that keeps track of what pixels have been explained and allows occlusion handling. In order to train the model we designed a principled loss function that accurately represents the properties of the instance segmentation problem. In the experiments carried out, we found that our method outperforms recent approaches on multiple person segmentation, and all state of the art approaches on the Plant Phenotyping dataset for leaf counting.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Object Detection with Transformers (2020, neutral): “Closest to our approach are end-to-end set predictions for object detection [43] and instance segmentation [41,30,36,42].”
Object-Centric Learning with Slot Attention (2020, background): “[73] Bernardino Romera-Paredes and Philip Hilaire Sean Torr.”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “Similarly to other works [36, 34], we use the soft intersection over union loss (sIoU) as the cost function between the mask predicted by our network m1 and the ground truth mask m2:”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “As a result, [38] proposed a Recurrent Neural Network (RNN) for this task.”
Shape-aware Instance Segmentation (2016, background): “Finally, in [26], a recurrent neural network was proposed to segment an image instance-by-instance.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “Current approaches generally use complex pipelines to handle instance extraction involving object proposals [19, 21, 7], conditional random ﬁelds (CRF) [30, 31], large recurrent neural networks (RNN) [23, 22, 2], or template matching methods [26].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “In [44] an LSTM-based [24] approach is proposed.”
Pix2seq: A Language Modeling Framework for Object Detection (2021, background): “Our work is also related to recurrent models in object detection (Stewart et al., 2016; Park & Berg, 2015; Romera-Paredes & Torr, 2016; Salvador et al., 2017; Ren & Zemel, 2017), in which the system learns to predict one object at a time.”
Path Aggregation Network for Instance Segmentation (2018, methodology): “RNN was utilized in [53, 50] to propose one instance in each time step.”
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, background): “This led to improved results in human pose estimation [17, 8] and semantic instance segmentation [22].”
Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling (2016, background): “Using a long short-term memory (LSTM) network [15], instance segmentations can be sequentially sampled [26].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “For example, [63, 61] propose recurrent neural networks to sequentially segment an instance at a time.”
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs (2015, background): “Parallel to our work, [23] proposes a recurrent neural net to label object instances sequential by keeping a memory of which pixels have been labeled so far.”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, methodology): “[31] propose an end-to-end recurrent network with convolutional LSTMs that sequentially outputs binary segmentation maps for each instance.”
Recurrent Pixel Embedding for Instance Grouping (2017, background): “Alternately, recurrent models that sequentially produce a list of instances [76, 73] offer another approach to address variable sized output structures in a uniﬁed manner.”
Bottom-up Instance Segmentation using Deep Higher-Order CRFs (2016, methodology): “An alternate method was proposed by [29] where a Recurrent Neural Network outputs an object instance at each time step.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, background): “Recurrent Neural Networks (RNNs) is adopted in several approaches [47, 45] to generate one instance mask at each time.”
End-to-End Instance Segmentation and Counting with Recurrent Attention (2016, methodology): “we compute a maximum-weighted bipartite graph matching between the output instances and ground-truth instances [22] and [18].”
One-Shot Instance Segmentation (2018, methodology): “ semantic segmentation [49, 55, 73, 60, 90, 9, 15, 48], where every pixel is classiﬁed into a category, instance segmentation additionally requires to discriminate between individual object instances [27, 18, 28, 62, 19, 39, 63, 72, 5, 14, 23, 29, 45, 70, 37]. Most current state-of-the-art systems a”