Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations

By Fenglin Liu, Yuanxin Liu, Xuancheng Ren, Xiaodong He, Xu Sun. 2019

96 citations recorded by the source corpus.

In vision-and-language grounding problems, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts, reflecting certain semantics. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on two representative vision-and-language grounding tasks, i.e., image captioning and visual question answering. In both tasks, the semantic-grounded image representations consistently boost the performance of the baseline models under all metrics across the board. The results demonstrate that our approach is effective and generalizes well to a wide range of models for image-related applications. (The code is available at \url{https://github.com/fenglinliu98/MIA)

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al., 2020; Chen et al., 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, background): “The attention mechanism allows probabilistic many-to-many relations instead of monotonic relations, as in [41, 37, 26].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[87] is based on an attention module to align grid or detection fea-”
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022, background): “Many propose to learn visual-semantic embedding for vision-text retrieval (Liu et al., 2019; Wu et al., 2019; Lu et al., 2019; Huang et al., 2020; Chen et al., 2021) by attention or objection detection models; and by vision-text contrastive learning (Zhang et al., 2020; Jia et al., 2021; Yuan et…”
Combined Scaling for Open-Vocabulary Image Classiﬁcation (2022, unknown): “…Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al., 2020; Chen et al., 2021).”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, background): “Given the enormous flexibility of the transformer encoder architecture, someworks [17, 30, 33, 38] tried to apply the attention mechanism of the transformer encoder architecture to process visual inputs and natural language together.”