A comparative evaluation of interactive segmentation algorithms

By Kevin McGuinness, Noel Edward O'Connor. 2010

344 citations recorded by the source corpus.

In this paper we present a comparative evaluation of four popular interactive segmentation algorithms. The evaluation was carried out as a series of user-experiments, in which participants were tasked with extracting 100 objects from a common dataset: 25 with each algorithm, constrained within a time limit of 2min for each object. To facilitate the experiments, a ''scribble-driven'' segmentation tool was developed to enable interactive image segmentation by simply marking areas of foreground and background with the mouse. As the participants refined and improved their respective segmentations, the corresponding updated segmentation mask was stored along with the elapsed time. We then collected and evaluated each recorded mask against a manually segmented ground truth, thus allowing us to gauge segmentation accuracy over time. Two benchmarks were used for the evaluation: the well-known Jaccard index for measuring object accuracy, and a new fuzzy metric, proposed in this paper, designed for measuring boundary accuracy. Analysis of the experimental results demonstrates the effectiveness of the suggested measures and provides valuable insights into the performance and characteristics of the evaluated algorithms.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “We adopt the test subset of Berkeley [46] introduced in [47], which consists of 100 masks for 96 images.”
Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “This is partly a consequence of existing benchmarks having set low accuracy targets for the task [6]: in the Berkeley benchmark[15] the threshold is set to 90% accuracy, 90% for the GrabCut dataset [16] and only 85% for PASCAL [5].”
Conditional Diffusion for Interactive Segmentation (2021, methodology): “GrabCut Berkeley SBD DAVIS Method NoC@85 NoC@90 NoC@90 NoC@85 NoC@90 NoC@85 NoC@90 Graph cut [2] 7.”
Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation (2015, background): “There are already some surveys [69], [70], [71] on interactive segmentation techniques, and thus this paper will act as a supplementary to them, where we discuss recent inﬂuential literature not covered by the previous surveys.”
Click Carving: Segmenting Objects in Video with Point Clicks (2016, methodology): “We use the timing data from the detailed study by (McGuinness and OConnor 2010), who find it takes a human on average 66.”
An interactive tool for semi-automated leaf annotation (2015, background): “Interactive segmentation is an active field of research, with solutions rooted in region growing methods, level set and live-wire approaches [2, 8, 23].”
F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (2020, methodology): “For the Berkeley dataset, we use the same test set as in [24], which includes 96 images with 100 object masks for testing.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “With the combination we achieve state-of-the-art on four standard interactive segmentation datasets: PASCAL VOC12 [20], GrabCut [46], DAVIS [40] and Berkeley [37].”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, methodology): “We conduct the interactive segmentation experiments on three public benchmarks with instance object annotations: PASCAL VOC [17], Grabcut [49], Berkeley [43].”
A survey of graph theoretical approaches to image segmentation (2013, background): “Some extensive studies [109,110] have been made to evaluate interactive segmentation methods.”
Secrets of GrabCut and Kernel K-Means (2015, background): “Error rates (%) are averaged over 82 images from Berkeley: 100 in [23] minus 18 images with multiple “identical” objects.”
Content-Aware Multi-Level Guidance for Interactive Instance Segmentation (2019, methodology): “For GrabCut and Berkeley, we run MCG [43] on the ‘accurate’ setting to obtain our set of object proposals.”
Regional Interactive Image Segmentation Networks (2017, neutral): “Berkeley dataset [19]: This dataset contains 100 singleobject images chosen from the Berkeley dataset to represent various challenges encountered in interactive segmentation, including similar foreground/background appearances, texture, existence of multiple similar objects, etc.”
A Fully Convolutional Two-Stream Fusion Network for Interactive Image Segmentation (2018, methodology): “Speciﬁcally, compared to RIS-Net, FCTSFN needs fewer clicks to achieve a certain IoU on Pascal VOC 2012 and Grabcut datasets; it needs 0.46 more clicks than RIS-Net to achieve 90% IoU on Berkeley dataset.”
MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation (2019, methodology): “In the general Semantic Segmentation problem, the objective is partitioning an image into a set of non-overlapping regions, which allows the homogeneous pixels to be clustered together [2].”