Learning to Refine Object Segments

By Pedro H. O. Pinheiro, Tsung-Yi Lin, Ronan Collobert, Piotr Doll'ar. 2016

857 citations recorded by the source corpus.

Object segmentation requires both object-level information and low-level pixel data. This presents a challenge for feedforward networks: lower layers in convolutional nets capture rich spatial information, while upper layers encode object-level knowledge but are invariant to factors such as pose and appearance. In this work we propose to augment feedforward nets for object segmentation with a novel top-down refinement approach. The resulting bottom-up/top-down architecture is capable of efficiently generating high-fidelity object masks. Similarly to skip connections, our approach leverages features at all layers of the net. Unlike skip connections, our approach does not attempt to output independent predictions at each layer. Instead, we first output a coarse ‘mask encoding’ in a feedforward pass, then refine this mask encoding in a top-down pass utilizing features at successively lower layers. The approach is simple, fast, and effective. Building on the recent DeepMask network for generating object proposals, we show accuracy improvements of 10–20% in average recall for various setups. Additionally, by optimizing the overall network architecture, our approach, which we call SharpMask, is 50 % faster than the original DeepMask network (under .8 s per image).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mask R-CNN (2017, methodology): “DeepMask [32] and following works [33, 8] learn to propose segment candidates, which are then classified by Fast R-CNN.”
Learning Features by Watching Objects Move (2016, methodology): “Note that a similar training setup was used for recent segmentation proposal methods [37, 38].”
Feature Pyramid Networks for Object Detection (2016, methodology): “There are recent methods exploiting lateral/skip connections that associate low-level feature maps across resolutions and semantic levels, including U-Net [31] and SharpMask [28] for segmentation, Recombinator networks [17] for face detection, and Stacked Hourglass networks [26] for keypoint estimat”
Panoptic Feature Pyramid Networks (2019, methodology): “As an alternative to dilation, an encoder-decoder [2] or ‘U-Net’ [49] architecture can be used to increase feature resolution [26, 44, 20, 47].”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “Recently, the task is achieved by deep learning approaches, such as DeepMask [32] and SharpMask [33].”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “Another related approach [11, 49] is to use segment-based region proposals [9, 35, 36] instead of box-based proposals.”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, background): “[39] propose refinement modules that have top-down and lateral connections.”
Shape-aware Instance Segmentation (2016, methodology): “In particular, we report the results of MCG [1], Deep-Mask [23] and SharpMask [24] using the publicly available pre-computed segmentation proposals.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “Current approaches generally use complex pipelines to handle instance extraction involving object proposals [20, 22, 7], conditional random fields (CRF) [32, 33], large recurrent neural networks (RNN) [24, 23, 2], or template matching [28].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Several methods produce proposals for instance segmentations and combine them, based on learned scores [33, 41, 42] or generate parts of instances and then combine them [14, 38].”
EDTER: Edge Detection with Transformer (2022, background): “Edge detection is one of the most fundamental problems in computer vision and has a wide variety of applications, such as image segmentation [8, 23, 39, 44, 45, 47], object detection [23], and video object segmentation [5, 57, 59].”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “These methods can either be a part of the network architecture [28, 29] or serve as a postprocessing step [30, 31].”
Iteratively Trained Interactive Segmentation (2018, methodology): “6, where we also added the result of the fully-automatic SharpMask [24] method for comparison.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “We then compare to two other baselines using SharpMask [30], the state-of-the-art object proposal technique.”
Focal Loss for Dense Object Detection (2017, background): “, Selective Search [12], EdgeBoxes [13], DeepMask [14], [15], RPN [3]) rapidly narrows down the number of can-”
Hybrid Task Cascade for Instance Segmentation (2019, methodology): “Many of these methods are based on CNN, including DeepMask [36], SharpMask [37], and InstanceFCN [10].”
A MultiPath Network for Object Detection (2016, methodology): “Modern object detectors predominantly follow the paradigm established by Girshick et al. in their seminal work on Region CNNs [11]: first an object proposal algorithm [17] generates candidate regions that may contain objects, second, a CNN classifies each proposal region.”
Path Aggregation Network for Instance Segmentation (2018, methodology): “Motivation Fully-connected layers, or MLP, were widely used in mask prediction in instance segmentation [10, 41, 34] and mask proposal generation [48, 49].”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Both instance and semantic segmentation has been approached using supervised [22, 25] and unsupervised learning [2, 9, 14, 10, 16, 21, 6] methods.”
DSSD : Deconvolutional Single Shot Detector (2017, background): “[22] who suggested that a factored version of the deconvolution module for a refinement network has the same accuracy as a more complicated one and the network will be more efficient.”