End-to-End Video Instance Segmentation with Transformers

By Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, Huaxia Xia. 2020

792 citations recorded by the source corpus.

Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches.Without bells and whistles, VisTR achieves the highest speed among all existing VIS models, and achieves the best result among methods using single model on the YouTube-VIS dataset. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy. We hope that VisTR can motivate future research for more video understanding tasks.Code is available at: https://git.io/VisTR

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

EDTER: Edge Detection with Transformer (2022, background): “Inspired by the recent success of vision transformers [9, 16, 61, 72], especially their capability of modeling longrange contextual information, we propose to tailor transformers for edge detection.”
Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “Built upon Transformers, VisTR (Wang et al., 2020) supervises and segments instances at the sequence level as a whole.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored applying transformers to various vision tasks: image classiﬁcation [5, 12], object detection [3, 61, 58, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [60, 53], and 3D point cloud processing [56].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
ViViT: A Video Vision Transformer (2021, background): “Although self-attention has been employed extensively in computer vision, it has, in contrast, been typically incorporated as a layer at the end or in the later stages of the network [75, 7, 32, 77, 83] or to augment residual blocks [30, 6, 9, 57] within a ResNet architecture [27].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Since its promise firstly demonstrated in Vision Transformer (ViT) [23], we have witnessed a flourish of full-Transformer models for image classification [57, 63, 67, 44, 80, 59], object detection [9, 91, 84, 20] and semantic segmentation [61, 65].”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “Besides, recent work also apply Transformers to various vision tasks, including object detection [5, 52], segmentation [41], image enhancement [6, 44] and video processing [47, 51].”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “Built upon Transformers, VisTR [35] views the VIS task as a parallel sequence prediction problem and segments instances at the sequence level.”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “The results thus inspire researchers to explore the applications of ViT on various vision tasks beyond image classiﬁcation, such as self-supervised learning [14, 9, 39], object detection [8, 85, 79, 18] and semantic segmentation [58, 62, 81].”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “Given an input image, AdaViT learns to adaptively derive policies on which patches , self-attention heads and transformer blocks to use or activate in the transformer backbone conditioned on the input image, encouraging us-ing less computation while maintaining the classiﬁcation accuracy.”
Do We Really Need Explicit Position Encodings for Vision Transformers? (2021, methodology): “Transformer reassures its advantage in instance segmentation [45], panoptic segmentation [43] and semantic segmentation [49].”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “VisTR [36] employed a direct end-to-end parallel sequence prediction framework to perform video instance segmentation.”
Language as Queries for Referring Video Object Segmentation (2022, methodology): “Beyond image ﬁeld, VisTR [50] extends the framework for video instance segmentation (VIS) [55] task and solves the problem in a direct end-to-end parallel sequence decoding manner.”