Learning Two-Branch Neural Networks for Image-Text Matching Tasks

By Liwei Wang, Yin Li, Jing Huang, Svetlana Lazebnik. 2017

537 citations recorded by the source corpus.

Image-language matching tasks have recently attracted a lot of attention in the computer vision field. These tasks include image-sentence matching, i.e., given an image query, retrieving relevant sentences and vice versa, and region-phrase matching or visual grounding, i.e., matching a phrase to relevant regions. This paper investigates two-branch neural networks for learning the similarity between these two data modalities. We propose two network structures that produce different output representations. The first one, referred to as an embedding network, learns an explicit shared latent embedding space with a maximum-margin ranking loss and novel neighborhood constraints. Compared to standard triplet sampling, we perform improved neighborhood sampling that takes neighborhood information into consideration while constructing mini-batches. The second network structure, referred to as a similarity network, fuses the two branches via element-wise product and is trained with regression loss to directly predict a similarity score. Extensive experiments show that our networks achieve high accuracies for phrase localization on the Flickr30K Entities dataset and for bi-directional image-sentence retrieval on Flickr30K and MSCOCO datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Examples of tasks that still represent a significant challenge for current artificial systems include text-to-video retrieval [25, 32, 54, 55, 63], text-based action or event localization [15], video captioning [36, 61], and video question answering [51, 63].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Ranking loss has been used for training the embedding in most recent works relating image and language modality for image-text retrieval [8, 10, 30, 40, 53].”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “Unlike the aforementioned works, another line of works focus on the way of learning visual representation from natural language supervisions, including both generative [5] and discriminative [48, 49, 58] methods.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “(ii) Another line of works focuses on learning transferable visual representation from natural language supervisions, including generative [12, 48] and contrastive methods [16,29,47,59,60,74].”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, background): “Most recent work on referring expressions uses CNN-LSTM based frameworks to model P (r|o) [19, 11, 32, 20, 18] or uses a joint vision-language embedding framework to model P (r, o) [22, 26, 27].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “To train the model, we use the bi-directional max-margin ranking loss [21, 22, 51, 52] as we would like to learn an embedding that works for both text-to-video and video-totext retrieval tasks.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Most recent works relating to text and imagemodality are trained with ranking loss [7, 9, 18, 23, 34].”
Position Focused Attention Network for Image-Text Matching (2019, methodology): “Therefore, many researchers have dedicated their efforts to study the relationship between the visual and the textual contents [Lee et al ., 2018; Gu et al ., 2018; Eisenschtat et al ., 2017; Ma et al ., 2015; Wang et al. , 2018; Klein et al. , 2015].”
Knowledge Aware Semantic Concept Expansion for Image-Text Matching (2019, methodology): “[Wang et al., 2018] introduced two-branch embeddings and considered the novel neighborhood constraints.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “Additionally, within-modal losses , also called structure preserving losses [19, 36], ensure that the neighbourhood structure within each modality is preserved in the newly built joint embedding space.”
VSE++: Improved Visual-Semantic Embeddings (2017, methodology): “Wang et al. (2017) propose an embedding network to fully replace the similarity measure used for the rank loss.”
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks (2022, background): “This task is also popular in recent V+L learning efforts [34, 49].”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, methodology): “Two-branch neural networks have been explored in various tasks for capturing cross-modality information (Nam, Ha, and Kim 2017; Wang et al. 2019) or different levels of information (Simonyan and Zisserman 2014; Wang et al.”
TransVG: End-to-End Visual Grounding with Transformers (2021, methodology): “…29.04 - Similarity Net [46] ResNet-101 34.54 60.89 CITE [37] ResNet-101 35.07 61.33 PIRC [24] ResNet-101 59.13 72.83 DDPN [61] ResNet-101 [46], CITE [37], DDPN [61], ZS-GNet [42], FAOA [56], and ReSC-Large [55] by a remarkable margin ( i.e. , 5.80% absolute improvement over the previous…”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “The goal of video-guided machine translation (Wang et al., 2019) is to translate a source language description into the target language using the video information as additional spatio-temporal context.”