Learning Unsupervised Video Object Segmentation Through Visual Attention

By Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen, Sanyuan Zhao, Steven C. H. Hoi, Haibin Ling. 2019

224 citations recorded by the source corpus.

This paper conducts a systematic study on the role of visual attention in Unsupervised Video Object Segmentation (UVOS) tasks. By elaborately annotating three popular video segmentation datasets (DAVIS, Youtube-Objects and SegTrack V2) with dynamic eye-tracking data in the UVOS setting, for the first time, we quantitatively verified the high consistency of visual attention behavior among human observers, and found strong correlation between human attention and explicit primary object judgements during dynamic, task-driven viewing. Such novel observations provide an in-depth insight into the underlying rationale behind UVOS. Inspired by these findings, we decouple UVOS into two sub-tasks: UVOS-driven Dynamic Visual Attention Prediction (DVAP) in spatiotemporal domain, and Attention-Guided Object Segmentation (AGOS) in spatial domain. Our UVOS solution enjoys three major merits: 1) modular training without using expensive video segmentation annotations, instead, using more affordable dynamic fixation data to train the initial video attention module and using existing fixation-segmentation paired static/image data to train the subsequent segmentation module; 2) comprehensive foreground understanding through multi-source learning; and 3) additional interpretability from the biologically-inspired and assessable attention. Experiments on popular benchmarks show that, even without using expensive video object mask annotations, our model achieves compelling performance in comparison with state-of-the-arts.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “Unsupervised VOS methods [33,56,50] aim to segment the primary objects in a video without the first frame annotations.”
JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection (2020, background): “Many SOD models have been developed under the assumption that the inputs are individual RGB/color images [21, 47, 66, 74–76] or sequences [56, 62, 67, 68].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “Unsupervised techniques [31, 7] perform segmentation without the guidance for foreground objects, and aim to estimate the object masks using salient features, independent motions, or known class labels automatically.”
Opening up Open World Tracking (2021, methodology): “5 J & F), outperforming several recent methods [2, 25, 72, 77, 81, 92], except UnOVOST, all fine-tuned for segmenting dominantly moving regions (67.”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “, AGS [69]) on DAVIS16 benchmark by a significant margin in terms of mean J (80.”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “popular benchmark datasets for video object segmentation tasks. It consists of 50 high-quality video sequences (30 for training and 20 for validation) in total. We follow the training strategy as AGS [56], COSNet [31], PDB [44] and MATNet [73] to use extra datasets. We use the image saliency dataset”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “Recent advances have been driven by deep learning techniques, from early, relatively simple architectures, such as recurrent network[45, 32, 63], and two-stream network [6, 49, 77], to recent, more powerful designs, such as teacher-student adaption [44], neural co-attention [26] and graph neural net”
Learning About Objects by Learning to Interact with Them (2020, methodology): “The supervised video object discovery approaches use different forms of supervision such as CNNs pre-trained for other tasks [35, 58, 57], supervised object proposal models [19, 11, 34], fixation data as annotation [61], or instance segmentation for single frames [46].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “ARP PDB MotAdapt LSMO AGS COSNet AGNN AnDiff Method [19] [42] [41] [46] [55] [24] [52] [68] Ours”
Human-Aware Motion Deblurring (2019, methodology): “Numerous CNN-based methods have also been applied in an endto-end fashion for image processing and generation problems, such as segmentation [24, 42], super-resolution [53, 22, 14, 15, 44], denoising [25, 49, 59, 47, 48], dehazing and deraining [32, 55], enhancement [31, 56] etc.”