SwiftNet: Real-time Video Object Segmentation
By Haochen Wang, Xiaolong Jiang, Haibing Ren, Yao Hu, Song Bai. 2021
187 citations recorded by the source corpus.
In this work we present SwiftNet for real-time semisupervised video object segmentation (one-shot VOS), which reports 77.8% $\mathcal{J}\& \mathcal{F}$ and 70 FPS on DAVIS 2017 validation dataset, leading all present solutions in overall accuracy and speed performance. We achieve this by elaborately compressing spatiotemporal redundancy in matching-based VOS via Pixel-Adaptive Memory (PAM). Temporally, PAM adaptively triggers memory updates on frames where objects display noteworthy inter-frame variations. Spatially, PAM selectively performs memory update and match on dynamic pixels while ignoring the static ones, significantly reducing redundant computations wasted on segmentation-irrelevant pixels. To promote efficient reference encoding, light-aggregation encoder is also introduced in SwiftNet deploying reversed sub-pixel. We hope SwiftNet could set a strong and efficient baseline for real-time VOS and facilitate its application in mobile vision. The source code of SwiftNet can be found at https://github.com/haochenheheda/SwiftNet.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Alan L. Yuille
- Ming-Hsuan Yang
- Chunhua Shen
- Dacheng Tao
- Philip H. S. Torr
- Xinggang Wang
- Xiang Bai
Citations in the corpus
- Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
- Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “(a) STM and variants (b) STCN (Ours) [18, 21, 22, 23, 24, 28, 29, 30] Figure 1: Left: The general framework of the popularly used Space-Time Memory (STM) networks, ignoring fine-level variations.”
- XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, background): “Particularly, SpaceTime Memory (STM) [36] is popular and has been extended by many follow-up works [43,8,18,54,50,31,9,44,33].”