Stacked Cross Attention for Image-Text Matching

By Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He. 2018

1,394 citations recorded by the source corpus.

In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuffs (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior works either simply aggregate the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or use a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in sentence as context and infer the image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% in text retrieval from image query, and 18.2% in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% and image retrieval by 16.6% (based on Recall@1 using the 5K test set).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, B2T2 (Alberti et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image- Setting dev test-P Triplet 72.76 73.55 Pair 75.37 75.97 Pair-biattn 77.14 77.87 Table 6: Experiments on three modified…”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “Following the splits in [35], we use 1,000 images for validation and test each and train on the rest.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Following the same split in [36], we use 1,000 images for validation and 1,000 images for testing and the rest for training.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “We did not use ranking losses [14,18], as we found that the binary classification loss works better, as reported in [28].”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, neutral): “SCAN[29] 48.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “For downstream tasks, such as image classi-ﬁcation, image retrieval (Karpathy and Fei-Fei 2015) (Lee et al. 2018) and object detection (Ren et al. 2015), the resulting models can extract feature representations for input images, which will be further used in following task-speciﬁc models.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Recently, focusing on hard-negatives has been shown to be effective in image-text embedding task for achieving high recall [8, 33, 60].”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “However, SCAN is much slower than FILIP (i.e., about 17ms slower per query) on MSCOCO with more (i.e., 5000) images because of the slower computation involved in the two-stage stacked cross-attention when computing the similarity.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “Unlike other works [2, 5, 9] which utilise high resolution regions extracted using a Faster-RCNN detector, our model is single stage and does not require any object detections.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Following existing work [44, 52], we use 1,000 images for validation, 1,000 images for testing, and use the rest of image-text pairs for model training.”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, neutral): “Recent work has extended this to vision and language tasks [2, 45–50], leading to compelling results in Visual Question Answering [4], Commonsense Reasoning [51], Natural Language Visual Reasoning [52], Entailment [53], Image-Text Retrieval [54, 55], Referring Expressions [56], and Vision-Language N”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The visual semantic embedding (VSE) models such as VSE++ (Faghri et al., 2017) and SCAN (Lee et al., 2018) belong to Figure 2a.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, background): “We evaluate our model on several downstream vision and language tasks, including Visual Question Answering (VQA) and Natural Language for Visual Rea-10 Model TR IR R@1 R@5 R@10 R@1 R@5 R@10 VSE++[10] 52.9 80.5 87.2 39.6 70.1 79.5 SCAN[17] 67.4 Natural Language for Visual Reasoning for Real Natural L”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “The traditional grid attention mechanism has been successfully used in modeling multi-level dependencies in image captioning [28], image question answering [30], text-to-image generation [29], unconditional image synthesis [31] and image-to-image translation [16], image/text retrieval [14].”
Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, background): “These mechanisms compete and attend to the input representation in a top-down fashion using crossattention [30, 29].”