Crossover Learning for Fast Online Video Instance Segmentation

By Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu. 2021

127 citations recorded by the source corpus.

Modeling temporal visual context across frames is critical for video instance segmentation (VIS) and other video understanding tasks. In this paper, we propose a fast on-line VIS model termed CrossVIS. For temporal information modeling in VIS, we present a novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames. Different from previous schemes, crossover learning does not require any additional network parameters for feature enhancement. By integrating with the instance segmentation loss, crossover learning enables efficient cross-frame instance-to-pixel relation learning and brings cost-free improvement during inference. Besides, a global balanced instance embedding branch is proposed for better and more stable online instance association. We conduct extensive experiments on three challenging VIS benchmarks, i.e., YouTube-VIS-2019, OVIS, and YouTube-VIS-2021 to evaluate our methods. CrossVIS achieves state-of-the-art online VIS performance and shows a decent trade-off between latency and accuracy. Code is available at https://github.com/hustvl/CrossVIS.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “CrossVIS [66] uses the instance feature in the current frame to localize the same instance in other frames.”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “We also evaluate nine open-source state-of-the-art methods (including FEELVOS [32], IoUTracker+ [39], MaskTrack R-CNN [39], SipMask [8], STEm-Seg [2], STMask [22], TraDeS [37], CrossVIS [40], and QueryInst [12]) on OVIS to analyze the OVIS dataset and serve as baselines for the OVIS challenge and fu”
Video Instance Segmentation using Inter-Frame Communication Transformers (2021, background): “In Table 2 (b), we refer the results reported in [3], which evaluated [1, 2] using ofﬁcial implementations.”