Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classiffication

By Max Bucher, Stéphane Herbin, Frédéric Jurie. 2016

220 citations recorded by the source corpus.

This paper addresses the task of zero-shot image classification. The key contribution of the proposed approach is to control the semantic embedding of images -- one of the main ingredients of zero-shot learning -- by formulating it as a metric learning problem. The optimized empirical criterion associates two types of sub-task constraints: metric discriminating capacity and accurate attribute prediction. This results in a novel expression of zero-shot learning not requiring the notion of class in the training phase: only pairs of image/attributes, augmented with a consistency indicator, are given as ground truth. At test time, the learned model can predict the consistency of a test image with a given set of attributes , allowing flexible ways to produce recognition inferences. Despite its simplicity, the proposed approach gives state-of-the-art results on four challenging datasets used for zero-shot recognition evaluation.

Subfields: Artificial Intelligence, Infectious Diseases, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Infectious Diseases

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning to Compare: Relation Network for Few-Shot Learning (2017, neutral): “5 SEC-ML [5] FV A 77.”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, background): “[32] embeds visual features into the attribute space, and then learns a metric to improve the consistency of the semantic embedding.”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, neutral): “compatibility learning frameworks [3, 4, 11, 15, 26, 31, 33, 39, 31, 12, 28, 1, 6, 23, 13, 20].”
Sampling Matters in Deep Embedding Learning (2017, background): “Models that transform images into rich, semantic representations lie at the heart of modern computer vision, with applications ranging from zero-shot learning [5, 41] and visual search [3, 10, 22, 29], to face recognition [6, 23, 25, 28] or fine-grained retrieval [22, 28, 29].”
Feature Generating Networks for Zero-Shot Learning (2017, background): “[50, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, methodology): “In the computer vision (CV) literature, deep metric learning (DML) has been used to great effect for learning visual representations [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34].”
A Simple Exponential Family Framework for Zero-Shot Learning (2017, methodology): “In another recent work, [4] proposed a model to improve the semantic embeddings using a metric learning formulation.”
Zero-Shot Semantic Segmentation (2019, methodology): “Early approaches address it as an embedding problem [1, 2, 6, 8, 16, 24, 31, 35, 38, 40, 41, 46].”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, unknown): “And the classic zero-shot image recognition techniques [1, 3, 6, 18, 29, 35, 47, 52, 54, 61, 65] are directly applied to learn from seen classes the visual-semantic mappings, which are then transferred to unseen ones.”
Learning Transferable Knowledge Through Embedding Spaces (2019, unknown): “57 Method SUN CUB AwA1 Romera-Paredes and Torr [18] 82.10 - 75.32 Zhang and Saligrama [19]† 82.5 30.41 76.33 Zhang and Saligrama [87]† 83.83 42.11 80.46 Bucher, Herbin, and Jurie [88]† 84.41 43.29 77.32 Xu et al. [89]† 83.5 53.6 84.5 Li et al. [90] † - 61.79 87.22 Ye and Guo [91]† 85.40 57.14 85.66 ”
Learning a Deep Embedding Model for Zero-Shot Learning (2016, methodology): “Inspired by humans’ ZSL ability, recently there is a surge of interest in machine ZSL [2, 47, 22, 1, 37, 43, 10, 31, 11, 14, 24, 46, 34, 4, 13, 3, 5, 48, 49].”
Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval (2017, background): “Many tasks require to jointly leverage images and natural text, such as zeroshot learning [4, 10], language generation [67, 35], multimedia retrieval [2, 3], image captioning [62, 16], and VQA [59, 45, 6].”
Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects (2019, methodology): “Method SJE [2] ESZSL [32] SynC [4] SEC [22] DEM [44] Ours Accuracy 50.1 47.2 54.5 43.3 58.3 59.4 Table 3: Experiments on the CUB image dataset is done in the baseline and our proposed methods, achieves a superior performance over that of projecting the feature space to the semantic space as in Chera”
Hard-Aware Deeply Cascaded Embedding (2016, background): “Deep metric embedding has attracted increasing attention for various tasks , such as visual product search [6, 18, 23, 27, 35], face recognition [21, 32, 41, 4], local image descriptor learning [9, 2, 12, 24], person/vehicle reidentification [22, 7, 43, 17], zero-shot image classification [19, 45, 5”
Gradient Matching Generative Networks for Zero-Shot Learning (2019, background): “Similarly, [37, 38, 39] learn a mapping from semantic embeddings to visual features, and, [40, 41, 42] learn a datadriven metric for comparing similarities between features and semantic embeddings.”
Generating Visual Representations for Zero-Shot Classification (2017, background): “, [2, 6, 5, 32] are focused on improving the embedding or the scoring function.”
Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning (2019, background): “[36], zero-shot learning [42, 1, 15], visual tracking [19, 31] and person re-identification [41, 9].”