SalGAN: Visual Saliency Prediction with Generative Adversarial Networks

By Junting Pan, Cristian Canton-Ferrer, Kevin McGuinness, Noel Edward O'Connor, Jordi Torres, Elisa Sayrol, Xavier Giró-i-Nieto. arXiv.org, 2017

432 citations recorded by the source corpus.

We introduce SalGAN, a deep convolutional neural network for visual saliency prediction trained with adversarial examples. The first stage of the network consists of a generator model whose weights are learned by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency maps. The resulting prediction is processed by a discriminator network trained to solve a binary classification task between the saliency maps generated by the generative stage and the ground truth ones. Our experiments show how adversarial training allows reaching state-of-the-art performance across different metrics when combined with a widely-used loss function like BCE. Our results can be reproduced with the source code and trained models available at https://imatge-upc.github. io/saliency-salgan-2017/.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/258fad95e709b6d0572ae6cc99efbbb14d32bdf2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Human Hands as Probes for Interactive Object Understanding (2021, methodology): “We compare to three classes of methods: a) objectness based approaches SalGAN [46] and DeepGaze2 [30], trained using human gaze data / manual labels; b) instance segmentation based approaches that use Mask RCNN [22] predicted masks for all / relevant classes; and c) interaction hotspots from…”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “…human pose synthesis [65, 66], stenographic applications [67 - 69], image manipulation applications [70 - 73, 234], visual saliency prediction [74 - 76], object detection [77 - 80], 3D image synthesis [81, 82], medical applications [86, 87], facial makeup transfer [94 - 96], facial landmark d…”
Generative Adversarial Networks: A Survey and Taxonomy (2019, background): “G ENERATIVE adversarial networks (GANs) are attracting growing interests in the deep learning community [1]–[6].”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “[34] who, following the large dissemination of Generative Adversarial Networks, trained their model by using adversarial examples.”
RGB-D salient object detection: A survey (2020, background): “Recently, generative adversarial networks (GANs) [163] have gained widespread attention for the salient detection task [164], [165].”
Self Adversarial Training for Human Pose Estimation (2017, background): “Another type of task is about generating heatmaps of labels as in semantic segmentation [26], saliency [29] or human pose estimation [10].”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, methodology): “[43] introduced SalGAN, a deep network for saliency prediction trained with adversarial examples.”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, background): “Generally, classification loss performs better than regression loss on many tasks such as image colorization [48], saliency prediction [29], and depth estimation [20].”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “Note that SALIENCY [40] produces only a single “importance” map (yellow).”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have been utilized in other image processing and computer vision tasks [354]–[357] such as object transfiguration [358], [359], semantic segmentation [360], visual saliency prediction [361], object tracking [362], [363], image dehazing [364]–[366], natural image matting [367], image inpainting ”
Revisiting Video Saliency Prediction in the Deep Learning Era (2021, methodology): “ND-avg represents the average results of three top-performing non-deep learning models: GBVS [27], Fang et al. [14] and ITTI [26]; D-avg indicates average results of three top-performing deep learning models: ACLNet, SalGAN [31] and DVA [29], according to Table 8.”