Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation

By Minghan Li, Shuai Li, Lida Li, Lei Zhang. 2021

61 citations recorded by the source corpus.

Modern one-stage video instance segmentation networks suffer from two limitations. First, convolutional features are neither aligned with anchor boxes nor with ground-truth bounding boxes, reducing the mask sensitivity to spatial location. Second, a video is directly divided into individual frames for frame-level instance segmentation, ignoring the temporal correlation between adjacent frames. To address these issues, we propose a simple yet effective one-stage video instance segmentation framework by spatial calibration and temporal fusion, namely STMask. To ensure spatial feature calibration with ground-truth bounding boxes, we first predict regressed bounding boxes around ground-truth bounding boxes, and extract features from them for frame-level instance segmentation. To further explore temporal correlation among video frames, we aggregate a temporal fusion module to infer instance masks from each frame to its adjacent frames, which helps our frame-work to handle challenging videos such as motion blur, partial occlusion and unusual object-to-camera poses. Experiments on the YouTube-VIS valid set show that the proposed STMask with ResNet-50/-101 backbone obtains 33.5 % / 36.8 % mask AP, while achieving 28.6 / 23.4 FPS on video instance segmentation. The code is released online https://github.com/MinghanLi/STMask.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “To dissect the OVIS dataset, we conduct a thorough evaluation of 9 state-of-the-art algorithms whose code is publicly available, including FEELVOS [51], IoUTracker+ [65], MaskTrack R-CNN [65], SipMask [7], STEm-Seg [2], STMask [31], TraDeS [59], CrossVIS [66], and QueryInst [14].”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “, STMask [22], STEm-Seg [2]) that achieve relatively higher performance on the heavily occluded object group, which demonstrate that the architecture or pipelines of these methods might more suitable to heavy occlusion perception.”