SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation

By Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao. 2020

195 citations recorded by the source corpus.

Single-stage instance segmentation approaches have recently gained popularity due to their speed and simplicity, but are still lagging behind in accuracy, compared to two-stage methods. We propose a fast single-stage instance segmentation method, called SipMask, that preserves instance-specific spatial information by separating mask prediction of an instance to different sub-regions of a detected bounding-box. Our main contribution is a novel light-weight spatial preservation (SP) module that generates a separate set of spatial coefficients for each sub-region within a bounding-box, leading to improved mask predictions. It also enables accurate delineation of spatially adjacent instances. Further, we introduce a mask alignment weighting loss and a feature alignment scheme to better correlate mask prediction with object detection. On COCO test-dev, our SipMask outperforms the existing single-stage methods. Compared to the state-of-the-art single-stage TensorMask, SipMask obtains an absolute gain of 1.0% (mask AP), while providing a four-fold speedup. In terms of real-time capabilities, SipMask outperforms YOLACT with an absolute gain of 3.0% (mask AP) under similar settings, while operating at comparable speed on a Titan Xp. We also evaluate our SipMask for real-time video instance segmentation, achieving promising results on YouTube-VIS dataset. The source code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “To dissect the OVIS dataset, we conduct a thorough evaluation of 5 state-of-the-art algorithms whose code is publicly available, including FEELVOS [38], IoUTracker+ [49], MaskTrack R-CNN [49], SipMask [4], and STEm-Seg [1].”
Video Instance Segmentation using Inter-Frame Communication Transformers (2021, background): “In Table 2 (b), we refer the results reported in [3], which evaluated [1, 2] using ofﬁcial implementations.”
OW-DETR: Open-world Detection Transformer (2021, background): “Several works have investigated the problem of standard object detection [2, 11, 13, 19, 25, 26, 29, 31].”
A Survey on Deep Learning Technique for Video Segmentation (2021, background): “[230], [232], [235], [236], [237], [252].”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “We also evaluate nine open-source state-of-the-art methods (including FEELVOS [32], IoUTracker+ [39], MaskTrack R-CNN [39], SipMask [8], STEm-Seg [2], STMask [22], TraDeS [37], CrossVIS [40], and QueryInst [12]) on OVIS to analyze the OVIS dataset and serve as baselines for the OVIS challenge and fu”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “And like image segmentation, most existing methods are specialist models comprising multiple stages with pipe-lined frameworks, e.g., track-detect [67, 57, 36, 6, 44], clip-match [3, 5], propose-reduce [37].”
Instances as Queries (2021, methodology): “3.6, QueryInst-VIS adopts the vanilla track method of MaskTrack R-CNN [57] and SipMask-VIS [6], while it obtains 4.3 AP improvement compared to MaskTrack R-CNN and 2.1 AP improvement compared to SipMask-VIS.”