What Do Different Evaluation Metrics Tell Us About Saliency Models?

By Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, Frédo Durand. 2016

932 citations recorded by the source corpus.

How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. The choice of evaluation metric depends on how saliency is defined and how the ground truth is represented. Metrics differ in how they rank saliency models, and this results from how false positives and false negatives are treated, whether viewing biases are accounted for, whether spatial deviations are factored in, and how the saliency maps are pre-processed. In this paper, we provide an analysis of 8 different evaluation metrics and their properties. With the help of systematic experiments and visualizations of metric computations, we add interpretability to saliency scores and more transparency to the evaluation of saliency models. Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.

Subfields: Computer Vision and Pattern Recognition, Sensory Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Sensory Systems

Citations in the corpus

Stacking with Auxiliary Features for Visual Question Answering (2018, positive): “Consistent with previous ﬁndings (Bylinskii et al., 2018), our results conﬁrm that EMD provides a ﬁner-grained comparison be-tween localization maps.”
Faster gaze prediction with dense networks and Fisher pruning (2018, methodology): “To verify that our models indeed perform close to the state of the art, we submitted saliency maps to the MIT Saliency Benchmark [3, 14, 4].”
Spatially Adaptive Computation Time for Residual Networks (2016, background): “Table 2 presents the AUC-Judd [5] metric, the area under the ROC-curve for the saliency map as a predictor for eye fixation positions.”
Using Human Gaze to Improve Robustness Against Irrelevant Objects in Robot Manipulation Tasks (2020, neutral): “Weevaluatedmodelstopredictsaliencymapsusingthe followingmultiplemetricsbecausethereisnoestablishedmetric [40],[41]:theareaundertheROCcurveproposedbyJudd (AUC-Judd)[42],Pearson’scorrelationcoefﬁcient(CC),nor-malizedscanpathsaliency(NSS)thatcomputestheaverageof…”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, background): “Different models may produce varying results for the same input, and there is no consensus on which model is the best [14].”
Evaluating and Aggregating Feature-based Model Explanations (2020, background): “Low complexity is loosely related to information gain from [ Bylinskii et al. , 2018 ] and to descriptive sparsity from [ War-necke et al. , 2019 ] .”
AGIL: Learning Attention from Human for Visuomotor Tasks (2018, methodology): “The ground truth saliency map is generated by converting discrete gaze positions to a continuous distribution using a Gaussian kernel with σ equals to 1 visual degree, as suggested by [3].”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, neutral): “Different models may produce varying results for the same input, and there is no consensus on which model is the best [100].”
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, background): “…of a post (Soleymani et al., 2017), or its perlocutionary force (how it is perceived by its audience), including aspects such as memorability (Khosla et al., 2015), saliency (Bylinskii et al., 2018), popularity (Khosla et al., 2014) and virality (Deza and Parikh, 2015; Alameda-Pineda et al., 2017).”
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, methodology): “Jetley et al. [11] also tested loss functions based on probability distances, such as X2 divergence, total variation distance, KL divergence and Bhattacharyya distance by considering saliency map models as generalized Bernoulli distributions.”
Predicting the Driver's Focus of Attention: The DR(eye)VE Project (2017, methodology): “Following the guidelines in [8], for the evaluation phase we rely on Pearson’s Fig.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “We direct interested readers to recent review papers on the topics of saliency evaluation metrics [Bylinskii et al. , 2019], saliency model performance analyses [Bylinskii et al. , 2016; He et al. , 2019a] and a closely related ﬁeld called salient object detection [Borji et al. , 2015].”
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “Based on the investigation in [53], [54], [55], we report quantitative evaluation results on five widely used evaluation metrics, including the traditional Area Under the ROC Curve (AUC), the shuffled AUC (sAUC), the Normalized Scanpath Saliency (NSS), the Similarity Metric (SIM) [56]”
Grounded Human-Object Interaction Hotspots From Video (2018, methodology): “We report error as KLDivergence, following [12], as well as other metrics (SIM, AUC-J) from the saliency literature [2].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “The AUC evaluates the ratio of ground-truth captured by the predicted affordance map under different thresholds [10].”