ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors

By Weicheng Kuo, Anelia Angelova, Jitendra Malik, Tsung-Yi Lin. 2019

128 citations recorded by the source corpus.

Instance segmentation aims to detect and segment individual objects in a scene. Most existing methods rely on precise mask annotations of every category. However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required. We introduce ShapeMask, which learns the intermediate concept of object shape to address the problem of generalization in instance segmentation to novel categories. ShapeMask starts with a bounding box detection and gradually refines it by first estimating the shape of the detected object through a collection of shape priors. Next, ShapeMask refines the coarse shape into an instance level mask by learning instance embeddings. The shape priors provide a strong cue for object-like prediction, and the instance embeddings model the instance specific appearance information. ShapeMask significantly outperforms the state-of-the-art by 6.4 and 3.8 AP when learning across categories, and obtains competitive performance in the fully supervised setting. It is also robust to inaccurate detections, decreased model capacity, and small training data. Moreover, it runs efficiently with 150ms inference time on a GPU and trains within 11 hours on TPUs. With a larger backbone model, ShapeMask increases the gap with state-of-the-art to 9.4 and 6.2 AP across categories. Code will be publicly available at: https://sites.google.com/view/shapemask/home.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Incremental Few-Shot Instance Segmentation (2021, neutral): “Approaches generally fall into two categories: grouping-based [4, 11, 17, 24] and proposal-based [2, 6, 15, 20] detection methods.”
Object-Contextual Representations for Semantic Segmentation (2019, background): “Various coarse-to-fine segmentation schemes have been developed [15,18,33,59,27,32,84] to gradually refine the segmentation maps from coarse to fine.”
OCNet: Object Context Network for Scene Parsing (2018, methodology): “, the shape of “bus” tends to be quadrilateral, pentagon, or hexagon under various views, thus, we might use a set of prior shape masks to refine the predictions like the recent ShapeMask (Kuo et al., 2019).”
Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling (2021, background): “Given bounding box annotations, [15, 20, 21, 30, 31] exploit pixelwise similarity to infer object masks while [18, 19, 32, 33] learn to share mask knowledge between mask and box supervision to enhance performances.”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “In order to extract object-centric spatial contexts from 2D images, we leverage a series of off-the-shelf expert models, including region proposal, region captioning [4], and semantic segmentation [39] modules to extract object-centric information.”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “Another family of methods are class-agnostic object proposal algorithms [46], [47], [48].”
PolyTransform: Deep Polygon Transformer for Instance Segmentation (2019, methodology): “[34] start with bounding boxes and refine them using shape priors.”
Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve (2020, methodology): “Recent approaches have focused on additionally predicting instance masks for each object [16,22].”
RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer (2020, methodology): “While segmentation models like Mask-RCNN [29] and ShapeMask [30] provide dense, pixel-level object supervision, it is practically easier and more efficient to label object detection data.”
OCNet: Object Context for Semantic Segmentation (2021, methodology): “…methods • use the shape structure information to regularize the segmentation, e.g., the shape of “bus” tends to be quadri-lateral, pentagon, or hexagon under various views, thus, we might use a set of prior shape masks to reﬁne the predictions like the recent ShapeMask (Kuo et al. 2019).”
SMIL: Multimodal Learning with Severely Missing Modality (2021, methodology): “Inspired by (Kuo et al. 2019), we approximate the missing modality using a weighted sum of modality priors learned from the modality-complete dataset.”
SegFix: Model-Agnostic Boundary Refinement for Segmentation (2020, background): “Refinement for Segmentation: Extensive studies [21,23,27,35,36] have proposed various mechanisms to refine the segmentation maps from coarse to fine.”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, background): “The fact is that feature subtraction is a different but also effective way to measure similarity between image features [1, 26].”