Similarity Comparisons for Interactive Fine-Grained Categorization

By Catherine Wah, Grant Van Horn, Steve Branson, Subhransu Maji, Pietro Perona, Serge J. Belongie. 2014

102 citations recorded by the source corpus.

Current human-in-the-loop fine-grained visual categorization systems depend on a predefined vocabulary of attributes and parts, usually determined by experts. In this work, we move away from that expert-driven and attribute-centric paradigm and present a novel interactive classification system that incorporates computer vision and perceptual similarity metrics in a unified framework. At test time, users are asked to judge relative similarity between a query image and various sets of images, these general queries do not require expert-defined terminology and are applicable to other domains and basic-level categories, enabling a flexible, efficient, and scalable system for fine-grained categorization with humans in the loop. Our system outperforms existing state-of-the-art systems for relevance feedback-based image retrieval as well as interactive classification, resulting in a reduction of up to 43% in the average number of questions needed to correctly classify an image.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “In contrast to our work, the oracle is often simulated with either a simple or no noise model [4, 32, 10, 55].”
Learning Concept Embeddings with Combined Human-Machine Expertise (2015, background): “Many researchers use similar embeddings to enhance the performance of classifiers [27, 9, 34], build retrieval sysFigure 1.”
Part-Stacked CNN for Fine-Grained Visual Categorization (2015, background): “Another line of research explored human-in-the-loop methods [8, 10, 45] to identify the most discriminative regions for classifying finegrained categories.”
We Don’t Need No Bounding-Boxes: Training Object Class Detectors Using Only Human Verification (2016, methodology): “Human-machine collaboration approaches have been successfully used in tasks that are currently too difficult to be solved by computer vision alone, such as fine-grained visual recognition [7, 11, 58, 59], semisupervised clustering [30], attribute-based image classification [6, 36, 37].”
Diversified Visual Attention Networks for Fine-Grained Object Classification (2016, background): “4) Human-in-the-Loop Approaches: Another branch is called human-in-the-loop methods [40]–[42], which need hu-”
Weakly Supervised Fine-Grained Image Categorization (2015, background): “Some other works [5, 27] recognize ﬁne-grained images with human in the loop.”
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020, background): “…to perform pairwise comparisons rather than providing direct labels (Simpson and Gurevych, 2018; Mailthody et al., 2019; Liang et al., 2018), including content search (Fürnkranz and Hüllermeier, 2010), image retrieval (Wah et al., 2014; Feng et al., 2019), and age estimation (Zhang et al., 2017).”
Crowdsourcing in Computer Vision (2016, unknown): “Welinder et al. [2010] are able to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that are different qualitatively.”
Fine-grained recognition without part annotations (2015, result): “the visipedia project [8, 43, 45], which is complementary to our approach.”
Embracing Error to Enable Rapid Crowdsourcing (2016, methodology): “In an effort to reduce cost, many previous papers have studied the tradeoffs between speed (cost) and accuracy on a wide range of tasks [68, 6, 67, 49].”