Associating Objects with Transformers for Video Object Segmentation

By Zongxin Yang, Yunchao Wei, Yi Yang. Neural Information Processing Systems, 2021

382 citations recorded by the source corpus.

This paper investigates how to realize better and more efficient embedding learning to tackle the semi-supervised video object segmentation under challenging multi-object scenarios. The state-of-the-art methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios, consuming multiple times computing resources. To solve the problem, we propose an Associating Objects with Transformers (AOT) approach to match and decode multiple objects uniformly. In detail, AOT employs an identification mechanism to associate multiple targets into the same high-dimensional embedding space. Thus, we can simultaneously process multiple objects' matching and segmentation decoding as efficiently as processing a single object. For sufficiently modeling multi-object association, a Long Short-Term Transformer is designed for constructing hierarchical matching and propagation. We conduct extensive experiments on both multi-object and single-object benchmarks to examine AOT variant networks with different complexities. Particularly, our R50-AOT-L outperforms all the state-of-the-art competitors on three popular benchmarks, i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%), while keeping more than $3\times$ faster multi-object run-time. Meanwhile, our AOT-T can maintain real-time multi-object speed on the above benchmarks. Based on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.

View Paper: https://www.semanticscholar.org/paper/ab0ef8e705fa88f079a055842ba20b4f413a69db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Kai Han
2. Yunhe Wang
3. Dacheng Tao
4. Chen Change Loy
5. Jianyuan Guo
6. Xiangtai Li
7. Ziwei Liu
8. Luc Van Gool

Citations in the corpus

• A Survey on Vision Transformer (2020, background): “[256] investigated how to realize better and more efficient embedding learning to tackle the semi-supervised video object segmentation under challenging”
• Transformer-Based Visual Segmentation: A Survey (2023, background): “Then, the following works [204] design a better memory-matching process. associating objects with transformers (AOT) [204] matches and decodes multiple objects jointly.”
• Track Anything: Segment Anything Meets Videos (2023, background): “2 AOT [15] NeurIPS2021 Mask One Pass 91.”
• AiATrack: Attention in Attention for Transformer Visual Tracking (2022, methodology): “We believe that the proposed AiA module could also be beneficial in other related tasks where the Transformer architecture can be applied to perform feature aggregation and information propagation, such as video object segmentation [60,33,15,39], video object detection [23] and multi-object tracking”
• XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, background): “YouTubeVOS 2019 val [15] Method G Js Fs Ju Fu FPSY18 CFBI [16] 81.0 80.6 85.1 75.2 83.0 3.4 SST [5] 81.8 80.9 - 76.7 - - MiVOS∗ [3] 82.4 80.6 84.7 78.1 86.4 - HMMN [12] 82.5 81.7 86.1 77.3 85.0 - CFBI+ [18] 82.6 81.7 86.2 77.1 85.2 4.0 STCN [4] 82.7 81.1 85.4 78.2 85.9 13.2 JOINT [8] 82.8 80.8 84.8 ”