Pixelwise Instance Segmentation with a Dynamically Instantiated Network

By Anurag Arnab, Philip H. S. Torr. 2017

242 citations recorded by the source corpus.

Semantic segmentation and object detection research have recently achieved rapid progress. However, the former task has no notion of different instances of the same object, and the latter operates at a coarse, bounding-box level. We propose an Instance Segmentation system that produces a segmentation map where each pixel is assigned an object class and instance identity label. Most approaches adapt object detectors to produce segments instead of boxes. In contrast, our method is based on an initial semantic segmentation module, which feeds into an instance subnetwork. This subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector, within an end-to-end CRF to predict instances. This part of our model is dynamically instantiated to produce a variable number of instances per image. Our end-to-end approach requires no post-processing and considers the image holistically, instead of processing independent proposals. Therefore, unlike some related work, a pixel cannot belong to multiple instances. Furthermore, far more precise segmentations are achieved, as shown by our substantial improvements at high APr thresholds.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Panoptic Feature Pyramid Networks (2019, methodology): “On Cityscapes, we compare Panoptic FPN with an approach for panoptic segmentation recently proposed in [1] in Table 4b.”
Panoptic Segmentation (2018, methodology): “Methods that generate non-overlapping instance segmentations [18, 3, 28, 2] use the same format as PS, but these methods typically only address thing classes.”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “Compared to more recent works [24, 4, 25, 5], our method falls behind”
Path Aggregation Network for Instance Segmentation (2018, background): “DIN [2] fused predictions from object detection and semantic segmentation systems.”
PointRend: Image Segmentation As Rendering (2019, background): “Alternative, bottom-up approaches group pixels to form object masks [28, 1, 22].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “[4, 5] apply higher-order Conditional Random Fields (CRFs) to refine the mask results.”
Attention-Guided Unified Network for Panoptic Segmentation (2018, methodology): “Moreover, segmentation-based methods aggregated pixel-level cues to compose instances combined with semantic segmentation [2, 26, 32] or depth ordering [44] results.”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, methodology): “In contrast, our method can handle such complex occlusions without problems as it treats the image holistically and learns to reason about occlusions, but does not employ a computationally expensive CRF like [5].”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “As a result, most recent successful approaches to instance segmentation have adopted more heuristic approaches that ﬁrst use an object detector to enumerate candidate instances and then perform pixel-level segmentation of each instance [57, 17, 55, 56, 2].”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “Another work [2, 32] combines the predictions of detection and semantic segmentation with a CRFasRNN [50] to generate instances.”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “Variant A is proposed in [26], whereas the mask-free parent of B and C is first described in [1] as the box consistency term.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “We use the architecture of Arnab et al. [16] as it produces both semantic-and instance-segmentations, and can be trained end-to-end, given object detections.”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “It segments an image into semantically meaningful instances [44], [45], [46], [47], [48].”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “Compared to instance segmentation in images [11, 16, 44, 1, 2, 22, 26], the problem considered in this paper requires not only to segment object instances in individual frames, but also to determine instance correspondences across multiple frames.”
Object Detection With Deep Learning: A Review (2018, background): “7) Mask R-CNN: Instance segmentation [96] is a challenging task which requires detecting all objects in an image and segmenting each instance (semantic segmentation [97]).”
BiSeg: Simultaneous Instance Segmentation and Semantic Segmentation with Fully Convolutional Networks (2017, methodology): “For example, the SBD dataset used in [2] is equivalent to the dataset used in our experiments.”