Static and space-time visual saliency detection by self-resemblance.

By Hae Jong Seo, Peyman Milanfar. Journal of Vision, 2009

700 citations recorded by the source corpus.

We present a novel unified framework for both static and space-time saliency detection. Our method is a bottom-up approach and computes so-called local regression kernels (i.e., local descriptors) from the given image (or a video), which measure the likeness of a pixel (or voxel) to its surroundings. Visual saliency is then computed using the said "self-resemblance" measure. The framework results in a saliency map where each pixel (or voxel) indicates the statistical likelihood of saliency of a feature matrix given its surrounding feature matrices. As a similarity measure, matrix cosine similarity (a generalization of cosine similarity) is employed. State of the art performance is demonstrated on commonly used human eye fixation data (static scenes (N. Bruce & J. Tsotsos, 2006) and dynamic scenes (L. Itti & P. Baldi, 2006)) and some psychological patterns.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/143973e23364190bc687fe694eb863417e8ba0ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Saliency-Aware Video Object Segmentation (2018, methodology): “Moreover, most video saliency methods simply treat the motion feature as another cue within their image saliency models [13], [14], [15], lacking an elegant framework to incorporate intra-frame and inter-frame information in a uniﬁed fashion.”
Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study (2013, background): “Le Meur: Le Meur et al. [58], [34] 2006 B X W × H 17 GBVS: Harel et al. [26] 2006 I M W × H 18 HouCVPR: Hou et al. [40] 2007 I M 64 × 64 19 Rarity-L: Local Rarity [42] 2007 I M W × H 20 Rarity-G: Global Rarity [42] 2007 I M W × H 21 HouNIPS: Hou et al. [41] 2008 I M W × H 22 Kootstra: Kootstra & Sho”
Visual saliency estimation by nonlinearly integrating features using region covariances. (2013, methodology): “…(excluding the task prior), Spectral residual (Hou & Zhang, 2007), SUN (Zhang et al., 2008), Attention based on information maximization (AIM) (Bruce & Tsotsos, 2009), Saliency detection by self-resemblance (Seo & Milanfar, 2009), and Context aware based saliency detection (Goferman et al., 2010).”
Gaze Prediction in Dynamic 360° Immersive Videos (2018, background): “Some work also attempts to use low-level appearance and motion cues as inputs or extend deep learning approaches to more complex scenarios such as stereo images [9] or videos [4] [7] [11] [31] [35] [32] [16] [12] [6] [18].”
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (2018, background): “Some of those studies [14, 41, 52] can be viewed as extensions of exiting static saliency models with additional motion features.”
Context-aware saliency detection (2010, methodology): “In [31], the resemblance between patches was also used as a cue for saliency estimation; however, there only local neighborhoods were compared.”
What Makes a Patch Distinct? (2013, background): “Note that computing the distance between every patch and the average patch bares some conceptual resemblance to the common approach of [5, 9, 19].”
Shifting More Attention to Video Salient Object Detection (2019, background): “Early VSOD models [8, 26, 28, 35, 52, 53, 62, 63, 74, 75] are built upon hand-crafted features (color, motion, etc .)”
State-of-the-Art in Visual Attention Modeling (2013, methodology): “Rosenholtz [191] assume a Gaussian and Seo and Milanfar [108] assume that P ðxÞ is a kernel density estimate (with the kernel that appears inside the summation on the denomi-”
A Benchmark of Computational Models of Saliency to Predict Human Fixations (2012, methodology): “Seo and Milanfar [66] use the same measurement protocol and compare their model against the same three above and the SUN model [87].”
Two-Stage Learning to Predict Human Eye Fixations via SDAEs (2016, methodology): “The AIM, SR, IS, QDCT, and SDSR algorithms look like to be influenced by the clustered background significantly.”
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “The heuristic saliency models [20], [21], [22], [23], [24] generally use hand-crafted features and design heuristic rules to perform visual saliency estimation in a bottom-up or topdown manner.”
Saliency-aware geodesic video object segmentation (2015, background): “Even though it is not the final goal of our proposed algorithm, we still evaluate the effectiveness of our approach by comparing our spatiotemporal saliency results against the stateof-art saliency methods [31, 13, 28, 14] on the SegTrack dataset [29].”
Saliency estimation using a non-parametric low-level vision model (2011, methodology): “Seo & Milanfar [14, 15] uses a self-resemblance mechanism to compute saliency, where a region with dissimilar curvature compared to its surroundings was designated as being highly salient.”
What Do Different Evaluation Metrics Tell Us About Saliency Models? (2016, methodology): “1b, include Achanta [2], AIM [11], CovSal [25], IttiKoch [44], [84], Judd [41], SR [73], Torralba [81], and WMAP [53].”
Region-Based Saliency Detection and Its Application in Object Recognition (2014, unknown): “algorithms have been proposed [36]–[38], [57], [68].”