Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks

By Long Chen, Hanwang Zhang, Jun Xiao, Wei David Liu, Shih-Fu Chang. 2017

312 citations recorded by the source corpus.

We propose a novel framework called Semantics-Preserving Adversarial Embedding Network (SP-AEN) for zero-shot visual recognition (ZSL), where test images and their classes are both unseen during training. SP-AEN aims to tackle the inherent problem - semantic loss - in the prevailing family of embedding-based ZSL, where some semantics would be discarded during training if they are non-discriminative for training classes, but could become critical for recognizing test classes. Specifically, SP-AEN prevents the semantic loss by introducing an independent visual-to-semantic space embedder which disentangles the semantic space into two subspaces for the two arguably conflicting objectives: classification and reconstruction. Through adversarial learning of the two subspaces, SP-AEN can transfer the semantics from the reconstructive subspace to the discriminative one, accomplishing the improved zero-shot recognition of unseen classes. Comparing

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Rectification-based Knowledge Retention for Continual Learning (2021, background): “The average H values are 10.2%, 36.73%, 38.03%, 36.73% and 42.48% for base, SFT, L1, L2 and LZSL [30] respectively.”
Adversarial Fine-Grained Composition Learning for Unseen Attribute-Object Recognition (2019, methodology): “Generally, most of the GAN-based ZSL methods [3, 5, 26, 8] inherit the inspiration of cGAN [17] which extends GAN from unsupervised learning into the semi-supervised setting by inputting the conditional variables along with the noise vectors.”
A Survey of Zero-Shot Learning (2019, methodology): “In [18], along with the zero-shot classifier, a reconstruction model is also learned.”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “SP-AEN [73] Hybrid Neural network ResNet [27] Attributes AWA, CUB, SUN, aPY ×”
Adaptive and Generative Zero-Shot Learning (2021, methodology): “Generative approaches (Chen et al., 2018; Felix et al., 2018; Kumar Verma et al., 2018; Mishra et al., 2018; Schonfeld et al., 2019; Paul et al., 2019; Xian et al., 2019) tackle the problem by synthesizing visual features of unseen classes from their semantic features with generative models like…”
Small Sample Learning in Big Data Era (2018, methodology): “…successfully applied to visual question answering(VQA) (Li et al., 2018c), semantic image segmentation (Luo et al., 2017a), image-to-image translation (Zhu et al., 2017b; Yi et al., 2017), zero-shot visual recognition (Chen et al., 2018b), and neural machine translation (He et al., 2016a, 2017a).”
Lifelong Zero-Shot Learning (2020, methodology): “Thus, the mainstream approach of ZSL methods [Chen et al. , 2018] is to construct the connection between visual space and semantic space.”
Region Graph Embedding Network for Zero-Shot Learning (2020, background): “Specifically, the embedding space can be categorized into three types: semantic space [23, 38, 48, 41, 9, 42, 37], feature space [5, 29, 40] and latent space [27, 59].”
Multi-modal Cycle-consistent Generalized Zero-Shot Learning (2018, background): “The majority of the ZSL and GZSL methods tend to compensate the lack of visual representation of the unseen classes with the learning of a mapping between visual and semantic spaces [16], [17].”
Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection (2019, background): “Motivated by these concerns, recent works have proposed to synthesize unseen examples by means of generative models such as autoencoders [4, 19], GANs and adversarial methods [49, 20, 42, 16, 23, 36] that take semantic vectors as input, and output images.”
A Variational Autoencoder with Deep Embedding Model for Generalized Zero-Shot Learning (2020, background): “2016), SAE (Kodirov, Xiang, and Gong 2017), DEM (Zhang, Xiang, and Gong 2017) and SP-AEN (Chen et al. 2018) learn a linear or nonlinear mapping between semantic space and image feature space.”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, background): “In Figure 4, we are comparing the results with a recent approach [7, 10, 53] that uses Improved-WGAN [15] for the same problem.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Although, most of the deep neural network models in this domain are trained using a discriminative loss function, a few generative models also exist [52, 55, 8] that are used as a data augmentation mechanism.”
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning (2019, methodology): “We conduct comparison with f-CLSWGAN [50], SE [41], SP-AEN [5] in this category.”
Natural and Effective Obfuscation by Head Inpainting (2017, background): “Prior work on structure-guided image generation has shown that such a guidance is indeed very helpful for generating images with complex inner structures (e.g. persons) [13, 5, 28, 30, 6, 33, 14, 2].”