Improving Pairwise Ranking for Multi-label Image Classification

By Yuncheng Li, Yale Song, Jiebo Luo. 2017

234 citations recorded by the source corpus.

Learning to rank has recently emerged as an attractive technique to train deep convolutional neural networks for various computer vision tasks. Pairwise ranking, in particular, has been successful in multi-label image classification, achieving state-of-the-art results on various benchmarks. However, most existing approaches use the hinge loss to train their models, which is non-smooth and thus is difficult to optimize especially with deep networks. Furthermore, they employ simple heuristics, such as top-k or thresholding, to determine which labels to include in the output from a ranked list of labels, which limits their use in the real-world setting. In this work, we propose two techniques to improve pairwise ranking based multi-label image classification by solving the aforementioned problems: (1) we propose a novel loss function for pairwise ranking, which is smooth everywhere, and (2) we incorporate a label decision module into the model, estimating the optimal confidence thresholds for each visual concept. We provide theoretical analyses of our loss function from the point of view of the Bayes consistency and risk minimization, and show its benefit over existing pairwise ranking formulations. We also demonstrate the effectiveness of our approach on two large-scale datasets, NUS-WIDE and MS-COCO, achieving the best reported result in the literature.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “2017 [29] VGG 3 77977 4104 40137 n/a 62.”
Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition (2019, background): “it is receiving increasing attention [17, 28, 34], since it underpins plenty of critical applications in content-based image retrieval and recommendation systems [4, 31].”
Learning to diagnose from scratch by exploiting dependencies among labels (2017, methodology): “…(e.g. Tsoumakas & Vlahavas (2007) and Read et al. (2008)), by training classiﬁers with loss functions that implicitly represent dependencies (e.g. Li et al. (2017)), and by using a sequence of single-label classiﬁers, each of which receives a subset of the previous predictions along with the…”
Attentive Recurrent Neural Network for Weak-supervised Multi-label Image Classification (2018, methodology): “To solve multi-label image classification, researchers [13, 18, 20, 21, 48] try to overcome two main challenges, (1) how to learn the discriminative feature representations and (2) how to effectively model the complex label relations.”
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning (2022, methodology): “We evaluate the performance of FSLL based on two different metrics: a bipartition-based metric, that is, F1-score [64], and a ranking-based metric, that is, area under curve (AUC), average precision (AP), Log loss [65], [66].”
Target-Aware Deep Tracking (2019, methodology): “For ease of implementation, we exploit a smooth approximated ranking loss [23] defined by”
Saliency-based Sequential Image Attention with Multiset Prediction (2017, methodology): “Evaluation Metrics To evaluate classification performance, macro-F1 and exact match (0-1) as defined in [26] are used.”
ELASTIC: Improving CNNs With Dynamic Scaling Policies (2018, methodology): “Following the literature of multilabel classification[41, 7, 39, 17], results are evaluated using macro/micro evaluations.”