Learning Robust Visual-Semantic Embeddings

By Yao-Hung Hubert Tsai, Ruslan Salakhutdinov. 2017

197 citations recorded by the source corpus.

Many of the existing methods for learning joint embedding of images and text use only supervised information from paired images and its textual attributes. Taking advantage of the recent success of unsupervised learning in deep neural networks, we propose an end-to-end learning framework that is able to extract more robust multi-modal representations across domains. The proposed method combines representation learning models (i.e., auto-encoders) together with cross-domain learning criteria (i.e., Maximum Mean Discrepancy loss) to learn joint embeddings for semantic and visual features. A novel technique of unsupervised-data adaptation inference is introduced to construct more comprehensive embeddings for both labeled and unlabeled data. We evaluate our method on Animals with Attributes and Caltech-UCSD Birds 200-2011 dataset with a wide range of applications, including zero and few-shot image recognition and retrieval, from inductive to transductive settings. Empirically, we show that our frame-work improves over the current state of the art on many of the considered tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Visual-Semantic Embedding: Joint visual-semantic models have shown excellent performance on several multimedia tasks, e.g., cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classification [10, 12, 20] video summarization [4, 41].”
Generalizing from a Few Examples (2019, background): “The two most popular applications are character recognition [14, 36, 37, 65, 70, 96, 113, 114, 119, 121, 130, 138, 146] and image classification [37, 70, 96, 106, 119, 121, 127, 130, 132, 138, 142, 143, 149].”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “ReViSE [34] uses auto-encoders to derive embeddings for images and words which allow them to leverage unlabeled data.”
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders (2018, methodology): “In contrast to [19] and [29], we also enforce a crossreconstruction loss, by decoding every encoded feature into every other modality.”
Adaptive Cross-Modal Few-Shot Learning (2019, background): “ReViSE [14] minimizes maximum mean discrepancy (MMD) of the distributions of the two representation spaces to align them.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Initially, most of the work was concentrated on combining low-level linguistic units such as words with images or videos for building visual-semantic embeddings (Frome et al., 2013; Kiros et al., 2014b; Liu et al., 2015; Cao et al., 2016; Tsai et al., 2017; Guo et al., 2018; Wang et al., 2019) usefu”
Zero-Shot Learning via Class-Conditioned Deep Generative Models (2017, background): “A few recent works exploited the idea of applying sematic regularization to the latent embedding spaced shared between encoder and decoder to make it suitable for ZSL tasks (Kodirov et al. 2017; Tsai et al. 2017).”
Relational Generalized Few-Shot Learning (2019, unknown): “, to regularize the feature space with textual embeddings for alignment on a distributional level [32], or to use attention mechanisms for synthesizing additional training examples for few-shot classes [33].”
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts (2017, methodology): “The majority of state-of-the-art approaches [2, 3, 36, 51, 41, 11, 42] consider zero-shot learning as a visual-semantic embedding Figure 1: Illustration of our proposed approach.”
Small Sample Learning in Big Data Era (2018, background): “Another attempt presenting an unsupervised-data adaptation inference framework into few/zero-shot learning was made by (Tsai et al., 2017) to learn robust visual-semantic embeddings.”
CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise (2017, methodology): “[30] trained visual-semantic embeddings with supervised and unsupervised objectives using labeled and unlabeled data to improve robustness of embeddings for transfer learning.”
Learning Factorized Multimodal Representations (2018, background): “Discriminative representation learning (Liang et al., 2018; Chen et al., 2017; Chaplot et al., 2017; Frome et al., 2013; Socher et al., 2013; Tsai et al., 2017) models the conditional distribution P ( Y ∣ X 1 ∶ M ) .”
Learning to Segment Every Thing (2017, unknown): “[35]).”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “Better representations could be learned for one-shot and few-shot image retrieval when correlated auto-encoders were used to reconstruct the representations for each modality [124].”
Improving One-Shot Learning through Fusing Side Information (2017, methodology): “Second, we provide the experiments for comparing the proposed method with direct side information fusion and ReViSE (Tsai et al., 2017).”
A Survey of Zero-Shot Learning (2019, background): “There are also some works on applying zero-shot learning methods to the paradigm of few-shot learning [2, 51, 66, 100, 138].”
Generalized Zero-Shot Learning with Deep Calibration Network (2018, background): “Another related setting is transductive zero-shot learning [18, 49] which exploits the unlabeled target data at training.”
Generalized Zero-Shot Learning via Aligned Variational Autoencoders (2019, background): “ReViSE [18] proposes a shared latent manifold learning using an autoencoder between the image features and class attributes.”
WildFish: A Large Benchmark for Fish Recognition in the Wild (2018, background): “Hence, many works have been proposed to learn the image-text embedding for multi-modal retrieval [7, 31, 41], zero/few-shot learning [2, 13, 30, 39], fine-grained recognition [2, 19, 31], and so on.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “Among early approaches of combining Vision & Semantics are tasks such as zero-shot learning where language/unseen descriptions of an unseen class are provided to describe it (e.g., [70,35,7,26,55,59,25,39,24,38,74,61]).”