Linking Image and Text with 2-Way Nets

By Aviv Eisenschtat, Lior Wolf. 2016

184 citations recorded by the source corpus.

Linking two data sources is a basic building block in numerous computer vision problems. Canonical Correlation Analysis (CCA) achieves this by utilizing a linear optimizer in order to maximize the correlation between the two views. Recent work makes use of non-linear models, including deep learning techniques, that optimize the CCA loss in some feature space. In this paper, we introduce a novel, bi-directional neural network architecture for the task of matching vectors from two data sources. Our approach employs two tied neural network channels that project the two views into a common, maximally correlated space using the Euclidean loss. We show a direct link between the correlation-based loss and Euclidean loss, enabling the use of Euclidean loss for correlation maximization. To overcome common Euclidean regression optimization problems, we modify well-known techniques to our problem, including batch normalization and dropout. We show state of the art results on a number of computer vision matching tasks including MNIST image matching and sentence-image matching on the Flickr8k, Flickr30k and COCO datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, neutral): “the proposed webly supervised approach, we divide the table in 3 rows (1.1-1.3). We compare our results with several representative image-text retrieval approaches, i.e., Embedding-Net [53], 2Way-Net [7], Sm-LSTM [19], OrderEmbedding [52], SAE [13], VSE [30] and VSEPP [8]. For these approaches, we d”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, methodology): “In the last few months, following the conclusion of our experiments, we have become aware of some new methods that achieve better Recall@1 [18].”
Position Focused Attention Network for Image-Text Matching (2019, background): “Therefore, many researchers have dedicated their efforts to study the relationship between the visual and the textual contents [Lee et al., 2018; Gu et al., 2018; Eisenschtat et al., 2017; Ma et al., 2015; Wang et al., 2018; Klein et al., 2015].”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, background): “Beyond the above, more research along this line focused on improving the visual or text embedding function (especially the aggregator), or designing auxiliary training objectives [8, 11, 17, 27, 33, 40, 41, 46].”
VSE++: Improved Visual-Semantic Embeddings (2017, methodology): “Klein et al. (2015) and Eisenschtat & Wolf (2016) use Canonical Correlation Analysis (CCA) to compute a linear projection for two views to a common space where the correlation of the transformed views is maximized.”
Visual Semantic Reasoning for Image-Text Matching (2019, neutral): “4 2WayNCVPR′17 [4] 55.”
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval (2019, background): “Text-image cross-modal retrieval has made great progress recently [16, 9, 22, 5, 4].”
Learning Semantic Concepts and Order for Image and Sentence Matching (2017, unknown): “0 2WayNet [4] 49.”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, background): “4 2 2WayNet [49] fixed VGG-16 feature from [17] 55.”
Dual-Path Convolutional Image-Text Embedding (2017, unknown): “9 2WayNet [2] fixed VGG-16 feature from [14] 49.”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, background): “Work in learning joint image-text spaces [3, 5, 10, 12, 20, 36] aims to project an image and a matching piece of text in such a way that the distance between related text and image is minimized.”
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives (2017, background): “(2016a)), zero-shot learning (Frome et al. (2013)), and modality conversion via synthesis (Reed et al. (2016b;a)). Such embeddings entail mappings from two (or more) domains into a common vector space in which semantically associated inputs (e.g., text and images) are mapped to similar locations. Th”
Multimodal One-shot Learning of Speech and Images (2018, methodology): “This is related to cross-modal retrieval tasks [35–37] used to evaluate multimodal networks.”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, methodology): “some point in the pipeline. Concerning image processing, the standard approach consists in using Convolutional Neural Networks (CNNs), usually pretrained on image classification tasks. In particular, [9, 15, 20, 29, 44] used VGGs, [10, 11, 16, 31] used ResNets. The problem with these kinds of CNNs i”
Graph Structured Network for Image-Text Matching (2020, background): “The main goal of global correspondence learning methods [19, 4, 29, 29, 31, 22, 34, 1] is to maximize similarity of matched image-text pairs.”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, methodology): “7 2WayNet [9] 55.”