Video Salient Object Detection via Fully Convolutional Networks

By Wenguan Wang, Jianbing Shen, Ling Shao. IEEE Transactions on Image Processing, 2017

608 citations recorded by the source corpus.

This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: 1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data and 2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without time-consuming optical flow computation. We further propose a novel data augmentation technique that simulates video training data from existing annotated image data sets, which enables our network to learn diverse saliency information and prevents overfitting with the limited number of training videos. Leveraging our synthetic video data (150K video sequences) and real videos, our deep video saliency model successfully learns both spatial and temporal saliency cues, thus producing accurate spatiotemporal saliency estimate. We advance the state-of-the-art on the densely annotated video segmentation data set (MAE of .06) and the Freiburg-Berkeley Motion Segmentation data set (MAE of .07), and do so with much improved speed (2 fps with all steps).

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/927e0a7b1f06102725a026f690077654fa53c76e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Stereo Video Object Segmentation Using Stereoscopic Foreground Trajectories (2019, background): “There is a recent interest in exploiting trajectories for representing long-term motion information [4], [5], [9], [11], [13], [40], [44].”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, methodology): “Quantitative results: As shown in Table 7, our model (without CRF binaryzation) outperforms previous VSOD models [64, 45, 80, 48, 79, 77, 16, 49] with human read-able attention maps.”
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (2018, background): “There are some salient object detection models [40, 1, 11, 61, 58, 60, 4, 62, 21] that attempt to uniformly highlight salient object regions in images or videos.”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “As there is no indication for objects to be segmented, conventional ZVOS methods resorted to certain heuristics, such as saliency [59, 62, 61, 7], object proposals [19, 37, 24], and discriminative motion patterns [31, 10, 33].”
RGB-D salient object detection: A survey (2020, background): “ys a key role in a range of real-world applications, such as stereo matching [11], image understanding [12], co-saliency detection [13], action recognition [14], video detection and segmentation [15]–[18], semantic segmentation [19], [20], medical image segmentation [21]–[23], object tracking [24], ”
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “To address the feature issue, deep saliency models [40]–[43] proposed to use feature representation learned from data by using Convolutional Neural Networks (CNNs) [44], [45].”
A Survey on Deep Learning Technique for Video Segmentation (2021, background): “images [72], [114] or optical flow fields [70], while several post-processing steps are still needed.”
Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection (2018, methodology): “We refer readers to [44] for more details.”
Deep Visual Attention Prediction (2017, background): “nal models. Since then, many followup works [5] have been proposed in this direction. In recent decades, there is a new wave in saliency detection [32], [3], [33], [34], [36], [37], [38], [39], [56], [49] that concentrated on uniformly highlighting the most salient object regions in an image, starti”
Salient Object Detection via Integrity Learning (2021, methodology): “[45] proposed a short-term analysis method for video frames by directly learning temporal saliency patterns from the adjacent frame pairs.”
Review of Visual Saliency Detection With Comprehensive Information (2018, methodology): “SG [100], STBP [102], SGSP [104]), and a deep learning based method (VFCN [107]) are used for comparison.”
A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping (2019, background): “Another related topic in parallel is salient object detection [35], [37], [39], [42], which can be dated to [36], [38] and has been extensively studied in the past decade.”
Revisiting Video Saliency Prediction in the Deep Learning Era (2021, background): “A related topic is salient object detection [63], [64] that uniformly highlights salient object regions in images [65], [66], [67], [68], [69], videos [70], [71], [72], [73], image/video groups [74], or RGBD data [75], [76].”
Shifting More Attention to Video Salient Object Detection (2019, methodology): “More recently, deep learning based VSOD models [31, 38, 39, 41, 67, 68, 76] have gained more attention inspired by the success of applying deep neural networks on image saliency detection [13–15,32,50,71,72,86,88–90,94].”
A Survey of Deep Learning-Based Object Detection (2019, background): “provide benchmarks for video salient object detection, and existing good algorithms [230] [231] [224] [232] [227] [233] [234] [235] [236] [237] [238][239][240][241] devote in this domain.”