Visual Saliency Transformer

By Nian Liu, Ni Zhang, Kaiyuan Wan, Junwei Han, Ling Shao. IEEE International Conference on Computer Vision, 2021

495 citations recorded by the source corpus.

Existing state-of-the-art saliency detection methods heavily rely on CNN-based architectures. Alternatively, we rethink this task from a convolution-free sequence-to-sequence perspective and predict saliency by modeling long-range dependencies, which can not be achieved by convolution. Specifically, we develop a novel unified model based on a pure transformer, namely, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD). It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Unlike conventional architectures used in Vision Transformer (ViT), we leverage multi-level token fusion and propose a new token upsampling method under the transformer framework to get high-resolution detection results. We also develop a token-based multi-task decoder to simultaneously perform saliency and boundary detection by introducing task-related tokens and a novel patch-task-attention mechanism. Experimental results show that our model outperforms existing methods on both RGB and RGB-D SOD benchmark datasets. Most importantly, our whole framework not only provides a new perspective for the SOD field but also shows a new paradigm for transformer-based dense prediction models. Code is available at https://github.com/nnizhang/VST.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/751b71158b7dcd2a7949e72a6ad8fb13657a401c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, background): “…to model long-range dependencies of long sequences in the ﬁeld of NLP. Recently, transformer models have been adapted to accommodate a wide variety of vision tasks [18], such as image classiﬁcation [10], saliency detection [27] and semantic segmentation [30], achieving promising performances.”
SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection (2022, methodology): “As described in the SOD review [1], SOD has been extended from RGB image [2]–[4] to RGB-D image [5], [6], a group of images [7], [8] and video [9], [10].”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “For example, some works [5, 82, 71, 55, 23, 41] model the vision task as a dictionary lookup problem with learnable queries, and use the Transformer decoder as a task-specific head on top of the CNN backbone.”
ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, methodology): “A token-based multi-task decoder [31] for RGB-D salient object detection method is developed by introducing task-related tokens and a novel patch-task-attention mechanism.”