Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation
By Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang. Neural Information Processing Systems, 2021
382 citations recorded by the source corpus.
This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without re-encoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/f3536a13c4bc97de0f703afbafb7318dce56e946
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Chen Change Loy
- Ping Luo
- Luc Van Gool
- Xiaogang Wang
- Ziwei Liu
- Dacheng Tao
- Ming-Hsuan Yang
- Alan L. Yuille
Citations in the corpus
- Language as Queries for Referring Video Object Segmentation (2022, background): “Most recent works [5, 33, 40, 48] lie in the group of matching-based methods, which perform feature matching to track the target objects.”
- Transformer-Based Visual Segmentation: A Survey (2023, background): “However, due to the extra datasets [352] they used, we do not list them here.”
- XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “For a fair comparison, we re-train the STCN [4] baseline under our setting.”
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “There are some prior works exploring memory models [59] in various vision tasks in videos, such as video object segmentation (VOS) [18,30,55,56], multi-object tracking (MOT) [2,14,29,73], visual object tracking (VOT) [38, 42, 77, 84], and action understanding [68].”