Performance evaluation in content-based image retrieval: overview and proposals

By Henning Müller, W. Müller, D. Squire, Stéphane Marchand-Maillet, Thierry Pun. Pattern Recognition Letters, 2001

658 citations recorded by the source corpus.

Abstract Evaluation of retrieval performance is a crucial problem in content-based image retrieval (CBIR). Many different methods for measuring the performance of a system have been created and used by researchers. This article discusses the advantages and shortcomings of the performance measures currently used. Problems such as defining a common image database for performance comparisons and a means of getting relevance judgments (or ground truth) for queries are explained. The relationship between CBIR and information retrieval (IR) is made clear, since IR researchers have decades of experience with the evaluation problem. Many of their solutions can be used for CBIR, despite the differences between the fields. Several methods used in text retrieval are explained. Proposals for performance measures and means of developing a standard test suite for CBIR, similar to that used in IR at the annual Text REtrieval Conference (TREC), are presented.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/90c7798c0a1168ff96e68f19e53ec733da74e2ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

Citations in the corpus

Features for image retrieval: an experimental comparison (2008, methodology): “To evaluate CBIR, several performance evaluation measures have been proposed [46] based on the precision P and the recall R:”
A survey of content-based image retrieval with high-level semantics (2007, methodology): “To evaluate CBIR, several performance evaluation measures have been proposed (Müller et al. 2001) based on the precision P and the recall R:”
Exploiting Unlabeled Data in Content-Based Image Retrieval (2004, methodology): “The evaluation measures used in Cbir have been aﬀected much by these used in common information retrieval [11].”
Content-based image retrieval using color difference histogram (2013, methodology): “In our experiments, we use the Precision and Recall curves, a performance metric commonly used in information retrieval [39,40].”
A review of content-based image retrieval systems in medical applications - clinical benefits and future directions (2004, methodology): “Only rarely are measurements used that are common to the domains of information retrieval [180] or content-based image retrieval [179] such as precision and recall defined as follows:”
A compact and efficient image retrieval approach based on border/interior pixel classification (2002, neutral): “[4, 7]).”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “Basically, the similarity between the query image and the database images is used to rank the database images in decreasing order of similarity [3].”
The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems (2006, methodology): “Standard datasets are vital for benchmarking the performance of information retrieval systems and allowing the comparison between different approaches or methods (Over et al., 2004; Müller et al., 2001; Narasimhalu et al., 1997; Smith, 1998).”
Visual Discovery at Pinterest (2017, background): “A visual search result is assumed to be relevant to a query image if the two images share the same class label, an approach that is commonly used for offline evaluation of visual search systems [30] in addition to human evaluation.”
Image retrieval: Ideas, influences, and trends of the new age (2008, background): “A comprehensive overview of benchmarking in CBIR can be found in [Muller et al. 2001].”
A Privacy-Preserving and Copy-Deterrence Content-Based Image Retrieval Scheme in Cloud Computing (2016, background): “In our experiments, the " precision " for a query is defined as that in [52]: P k = k /k, where k is the number of real similar images in the k retrieved images.”
Content-based image retrieval: approaches and trends of the new age (2005, positive): “The design of a CBIR benchmark requires careful design in order to capture the inherent subjectivity in image retrieval. One such proposal can be found in [ 75 ].”