Video Instance Segmentation using Inter-Frame Communication Transformers

By Sukjun Hwang, Miran Heo, Seoung Wug Oh, Seon Joo Kim. 2021

167 citations recorded by the source corpus.

We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality. In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip. Specifically, we propose to utilize concise memory tokens as a mean of conveying information as well as summarizing each frame scene. The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (AP 44.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). Our method can also be applied to near-online inference for processing a video in real-time with only a small delay. The code will be made available.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Xiaogang Wang
4. Alan L. Yuille
5. Chunhua Shen
6. Ping Luo
7. Ming-Hsuan Yang
8. Dacheng Tao

In Model Architectures

1. Alan L. Yuille
2. Ziwei Liu
3. Ping Luo
4. Philip H. S. Torr
5. Song Bai
6. Chen Change Loy
7. Xiang Bai
8. Kai Chen

Citations in the corpus

• Occluded Video Instance Segmentation: A Benchmark (2021, background): “IFC [32] further reduces the computations of full space-time transformers by only executing attention between memory tokens.”
• Transformer-Based Visual Segmentation: A Survey (2023, background): “IFC [170] adopts message tokens to exchange temporal context among different frames.”
• Language as Queries for Referring Video Object Segmentation (2022, background): “The recent success of Transformer [39] in object detection [3, 53] and video instance segmentation [14, 43, 44] demonstrates a promising solution.”