See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks

By Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, Fatih Murat Porikli. Computer Vision and Pattern Recognition, 2019

532 citations recorded by the source corpus.

We introduce a novel network, called as CO-attention Siamese Network (COSNet), to address the unsupervised video object segmentation task from a holistic view. We emphasize the importance of inherent correlation among video frames and incorporate a global co-attention mechanism to improve further the state-of-the-art deep learning based solutions that primarily focus on learning discriminative foreground representations over appearance and motion in short-term temporal segments. The co-attention layers in our network provide efficient and competent stages for capturing global correlations and scene context by jointly computing and appending co-attention responses into a joint feature space. We train COSNet with pairs of video frames, which naturally augments training data and allows increased learning capacity. During the segmentation stage, the co-attention model encodes useful information by processing multiple reference frames together, which is leveraged to infer the frequently reappearing and salient foreground objects better. We propose a unified and end-to-end trainable framework where different co-attention variants can be derived for mining the rich context within videos. Our extensive experiments over three large benchmarks manifest that COSNet outperforms the current alternatives by a large margin. We will publicly release our implementation and models.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e54fe27ed18d513e3e9171661bd9b6e1c982b7d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

An Empirical Study of Detection-Based Video Instance Segmentation (2019, background): “Many recent advances have been made in this challenging area ( e.g ., [1, 5, 17, 18, 25, 26]).”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “Considering their superior feature extraction capability [36, 14, 16], we instantiate both EnR and EnQ with ResNet-based [4] backbones.”
RGB-D salient object detection: A survey (2020, background): “ention mechanisms have been developed to explore the underlying correlations across multiple modalities, and are widely studied in visual question answering [182], [183] and video object segmentation [184]. Thus, for the RGB-D based SOD task, we could also develop attentionbased fusion algorithms to”
Learning Video Object Segmentation From Unlabeled Videos (2020, background): “Various effective networks have been explored, from some early, relatively simple architectures, such as recurrent network[45, 32, 63], and two-stream network [6, 49, 77], to recent, more powerful designs, such as teacher-student adaption [44], neural co-attention [26] and graph neural network[58, 6”
Human-Aware Motion Deblurring (2019, methodology): “Numerous CNN-based methods have also been applied in an endto-end fashion for image processing and generation problems, such as segmentation [24, 42], super-resolution [53, 22, 14, 15, 44], denoising [25, 49, 59, 47, 48], dehazing and deraining [32, 55], enhancement [31, 56] etc.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “LTV FST COSEG ARP LVO PDB FSEG SFL MotAdapt LSMO AGS COSNet AGNN Method [30] [32] [47] [19] [45] [42] [16] [8] [41] [46] [55] [24] [52] Ours”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “extended [83] by making better use of information from multiple frames [89], [175], [176], encoding spatial context [105], and incorporating temporal consistency to improve representation power and computation efficiency [96], [104].”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “We follow the training strategy as AGS [56], COSNet [31], PDB [44] and MATNet [73] to use extra datasets.”
Few-Shot Semantic Segmentation with Democratic Attention Networks (2020, background): “Few shot learning has also been explored in video analysis [20, 21].”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, background): “Recently, with the renaissance of deep learning, more research efforts have been devoted to tackling this in deep learning frameworks, leading to a zero-shot solution [13, 21, 58, 7, 30, 31, 29, 37].”
Region Graph Embedding Network for Zero-Shot Learning (2020, methodology): “More recently, by pursing automatic part discovery [53], attention mechanisms [57, 56, 55, 25] have been applied into ZSL and GZSL [52, 80, 78, 30] for capturing multiple semantic regions, which can facilitate desirable knowledge transfer.”
PARE: Part Attention Regressor for 3D Human Body Estimation (2021, methodology): “Given the success of attention-based methods on other tasks [11, 18, 34, 55], we exploit insights gained from the occlusion sensitivity analysis to focus attention on body parts.”