Structure-Measure: A New Way to Evaluate Foreground Maps
By Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, Ali Borji. International Journal of Computer Vision, 2017
1,869 citations recorded by the source corpus.
Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the field of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several measures (e.g., area-under-the-curve, F1-measure, average precision, etc.) have been used to evaluate the similarity between a foreground map and a ground-truth map. The existing measures are based on pixel-wise errors and often ignore the structural similarities. Behavioral vision studies, however, have shown that the human visual system is highly sensitive to structures in scenes. Here, we propose a novel, efficient (0.005 s per image), and easy to calculate measure known as S-measure (structural measure) to evaluate foreground maps. Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a foreground map and a ground-truth map. We demonstrate superiority of our measure over existing ones using 4 meta-measures on 5 widely-used benchmark datasets. Furthermore, we conduct a behavioral judgment study over a new database. Data from 45 subjects shows that on average they preferred the saliency maps chosen by our measure over the saliency maps chosen by the state-of-the-art measures. Our experimental results offer new insights into foreground map evaluation where current measures fail to truly examine the strengths and weaknesses of models. Code: https://github.com/DengPingFan/S-measure.
Subfields: Computer Vision and Pattern Recognition, Sensory Systems
View Paper: https://www.semanticscholar.org/paper/381d56f925530bc0fcd11e7cd4c4f7bb033fe000
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Ling Shao
- Luc Van Gool
- Ming-Ming Cheng
- Deng-Ping Fan
- Ming-Hsuan Yang
- Dacheng Tao
- Jianbing Shen
- Huchuan Lu
In Sensory Systems
Citations in the corpus
- Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection (2019, methodology): “It has a wide range of applications, including video/image segmentation [17, 40], object recognition [46], visual tracking [3], foreground maps evaluation [14, 15], image retrieval [6,16,22,38], content-aware image editing [8], information discovery [58], photo synthesis [5, 29], and weakly supervis”
- BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, methodology): “We adopt 4 widely used metrics including S-measure (Sα) [17], maximum E-measure (Eξ) [18], maximum F-measure (Fβ) [1], Mean Absolute Error (MAE).”
- Stacked Cross Refinement Network for Edge-Aware Salient Object Detection (2019, background): “5 ) [9] are also adopted to evaluate saliency maps.”
- Camouflaged Object Segmentation with Distraction Mining (2021, methodology): “We use four widely used and standard metrics to evaluate our method: structure-measure ( S α ) [10], adaptive E-measure ( E adφ ) [11], weighted F-measure ( F wβ ) [35], and mean absolute error ( M ).”
- UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders (2020, methodology): “Evaluation Metrics: Four evaluation metrics are used, including two widely used: 1) Mean Absolute Error (MAE M); 2) mean F-measure (Fβ) and two recently proposed: 3) Enhanced alignment measure (mean E-measure, Eξ) [15] and 4) Structure measure (S-measure, Sα) [14].”
- Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, background): “Motivated by the spatial envelope [52] and the block-level strategy [18, 19], we then split the image into different scales ( i.e ., 1 × 1, 2 × 2, . . . , 5 × 5).”
- Cross-level Feature Aggregation Network for Polyp Segmentation (2023, methodology): “Additionally, four metrics are introduced, which are widely used in the field of object detection [32,52,53] , including S-measure ( S α ) [54] , F-measure [55] ( F w β ), E φ [56] , and mean absolute error ( M ) [57] .”
- Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images (2020, background): “5) suggested in the original paper [62].”
- Shifting More Attention to Video Salient Object Detection (2019, methodology): “To quantitatively assess the model performance, we adopt 2 popular evaluation metrics: Mean Absolute Error ( MAE ) M [58], F-measure F [1], and the recent released structure measure S-measure S [18].”
- SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection (2022, background): “They are the precision-recall (PR) curve [83], S-measure [84], F-measure [85], E-measure [86] and mean absolute error (MAE) [87].”
- Attentive Feedback Network for Boundary-Aware Salient Object Detection (2019, background): “[8], and it can be used to evaluate non-binary foreground maps.”
- Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, methodology): “• Structural measure (S-measure) [128], different from the above metrics which only address pixel-wise errors, evaluates structural similarity between the real-valued saliency map and the binary ground-truth.”
- Visual Saliency Transformer (2021, background): “Specifically, Structure-measure Sm [13] evaluates region-aware and object-aware structural similarity.”
- Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection (2019, methodology): “For comprehensively evaluating various methods, we adopt five evaluation metrics including precision-recall (PR) curve, F-measure (Fβ) [1], mean absolute error (MAE) [2] and recently proposed S-measure (Sλ) [12] and E-measure (Eγ) [13].”
- Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “5 [14].”
- F3Net: Fusion, Feedback and Focus for Salient Object Detection (2019, background): “5 ) (Fan et al. 2017) and E-measure ( E ξ ) (Fan et al. 2018) are also widely used to evaluate salient maps.”
- CaraNet: context axial reverse attention network for segmentation of small medical objects (2021, positive): “In addition to mean Dice and mean IoU, we also apply four other measurement metrics: weighted dice metric 𝐹 𝛽𝑤 , MAE, enhanced alignment metric Ε 𝜙𝑚𝑎𝑥 [45], and structural measurement 𝑆 𝛼 [46].”
- Efficient and Accurate Arbitrary-Shaped Text Detection With Pixel Aggregation Network (2019, background): “In virtue of recent development of object detection [41, 30, 11, 9, 5, 8, 56] and segmentation [1, 55, 33, 53, 6, 7] based on CNN [15, 20, 47, 25], scene text detection has witnessed great progress [46, 42, 57, 31, 35, 49, 24].”
- Weakly-Supervised Salient Object Detection via Scribble Annotations (2020, background): “Existing saliency evaluation metrics (Mean Abosolute Error, Precision-recall curves, F-measure, E-measure [7] and S-measure [6]) focus on measuring accuracy of the prediction, while neglect whether a predicted saliency map complies with human perception or not.”