Valmet: A New Validation Tool for Assessing and Improving 3D Object Segmentation

By Guido Gerig, Matthieu Jomier, M. Chakos. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2001

388 citations recorded by the source corpus.

Extracting 3D structures from volumetric images like MRI or CT is becoming a routine process for diagnosis based on quantitation, for radiotherapy planning, for surgical planning and image-guided intervention, for studying neurodevelopmental and neurodegenerative aspects of brain diseases, and for clinical drug trials. Key issues for segmenting anatomical objects from 3D medical images are validity and reliability. We have developed VALMET, a new tool for validation and comparison of object segmentation. New features not available in commercial and public-domain image processing packages are the choice between different metrics to describe differences between segmentations and the use of graphical overlay and 3D display for visual assessment of the locality and magnitude of segmentation variability. Input to the tool are an original 3D image (MRI, CT, ultrasound), and a series of segmentations either generated by several human raters and/or by automatic methods (machine). Quantitative evaluation includes intra-class correlation of resulting volumes and four different shape distance metrics, a) percentage overlap of segmented structures (R intersect S)/(R union S), b) probabilistic overlap measure for non-binary segmentations, c) mean/median absolute distances between object surfaces, and maximum (Hausdorff) distance. All these measures are calculated for arbitrarily selected 2D cross-sections and full 3D segmentations. Segmentation results are overlaid onto the original image data for visual comparison. A 3D graphical display of the segmented organ is color-coded depending on the selected metric for measuring segmentation difference. The new tool is in routine use for intra- and inter-rater reliability studies and for testing novel automatic machine-segmentation versus a gold standard established by human experts. Preliminary studies showed that the new tool could significantly improve intra- and inter-rater reliability of hippocampus segmentation to achieve intra-class correlation coefficients significantly higher than published elsewhere.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7952038488d4bf8d6362a30dd7386c0c88f43902

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation (2010, methodology): “The segmentation results are evaluated by assigning a score to each test case on a variety of metrics [14].”
Dense image registration through MRFs and efficient linear programming (2008, methodology): “The quantitative results are presented in Table 2. Visual results of the surface distance (SD) for the gray and white matter of one of the data sets are shown in Fig. 4. The SD map is computed using the tool 6 described in Gerig et al. (2001) .”
3D Segmentation in the Clinic: A Grand Challenge (2007, background): “Outside the medical domain, there have been competitions for 6 7 8 constructing an accurate system to predict ratings 9 is attracting interest from thousands of research teams worldwide.”
Statistical validation of image segmentation quality based on a spatial overlap index. (2004, background): “Other validation metrics considered for statistical validation included Jaccard similarity coefficient (17), odds ratio (18), receiver operating characteristic analysis (19–22), mutual information (3,22), and distance-based statistics (23,24).”
Unbiased diffeomorphic atlas construction for computational anatomy. (2004, methodology): “We use a previously developed validation package VALMET (Gerig et al., 2001) that includes a probabilistic overlap measure between two fuzzy segmentations.”
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool (2015, methodology): “However these two measures are not common as evaluation measures of medical image segmentation because of their sensibility to segments size, i.e. they penalize errors in small segments more than in large segments [6, 8, 10].”
Learning computationally efficient approximations of complex image segmentation metrics (2013, methodology): “The Hausdorff distance [5] has been used in a variety of applications, including the validation of image segmentation results [7], [8].”
An Efficient Algorithm for Calculating the Exact Hausdorff Distance (2015, background): “Many researchers have noted the computational complexity of the HD [8], [11], [21].”
Toward automated segmentation of the pathological lung in CT (2005, methodology): “This was done by calculating the mean absolute distance [25] between the manual contours and the interpolated contours1.”
Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration (2009, methodology): “To better enable individuals to make these choices, the Valmet software tool (http://www.ia.unc.edu/public/valmet/) (Gerig et al., 2001) and the Non-rigid Image Registration Evaluation Project (NIREP) (http://www. nirep.org) were developed.”
Combination Strategies in Multi-Atlas Image Segmentation: Application to Brain MR Data (2009, methodology): “Apart from SI, the mean average surface distance (MASD) between the segmented structures was also evaluated [2], [ 31 ].”
Image analysis and superimposition of 3-dimensional cone-beam computed tomography models. (2006, methodology): “After registration, we can assess the overlay of the 3D models using Valmet.(23) Valmet(23) software allows visual and quantitative assessment of the location and magnitude of changes over time segmentation via graphic overlays and calculation of the distances between the surfaces of the 3D models a”
Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation (2004, neutral): “In many applications, assessment of boundary differences is useful and the Hausdorff measure and modifications have been used for this [16].”
Generalized Overlap Measures for Evaluation and Validation in Medical Image Analysis (2006, methodology): “[10]; some of these quantities can be computed using the Valmet software described in [11].”
Automated Volumetry and Regional Thickness Analysis of Hippocampal Subfields and Medial Temporal Cortical Structures in Mild Cognitive Impairment (2014, methodology): “Table VII compares the DSC reported for each subfield in the cross-validation study in Table III to the symmetric root mean square (RMS) surface distance metric, which measures average distance between automatic and manual segmentation surfaces (Gerig et al., 2001).”
A brain tumor segmentation framework based on outlier detection (2004, methodology): “We used the VALMET segmentation validation tool (Gerig et al., 2001) to generate five validation metrics.”