Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations

By Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, Wei-Ying Ma. 2019

199 citations recorded by the source corpus.

We propose the Unified Visual-Semantic Embeddings (Unified VSE) for learning a joint space of visual representation and textual semantics. The model unifies the embeddings of concepts at different levels: objects, attributes, relations, and full scenes. We view the sentential semantics as a combination of different semantic components such as objects and relations; their embeddings are aligned with different image regions. A contrastive learning approach is proposed for the effective learning of this fine-grained alignment from only image-caption pairs. We also present a simple yet effective approach that enforces the coverage of caption embeddings on the semantic components that appear in the sentence. We demonstrate that the Unified VSE outperforms baselines on cross-modal retrieval tasks; the enforcement of the semantic coverage improves the model's robustness in defending text-domain adversarial attacks. Moreover, our model empowers the use of visual cues to accurately resolve word dependencies in novel sentences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “As aforementioned in § 1, complex aggregators f have been investigated in the VSE literature [17, 27, 43, 45, 46], such as sequence-to-sequence encoder (Seq2Seq), graph convolution network (GCN), self-attention encoder (SelfAttn), etc.”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “Its performance may be boosted by considering structured representations of both images (e.g., Lu et al., 2016; Wu et al., 2019) and texts (Steedman, 2000).”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “[40] factorize sentence into objects, attributes, relations and sentences, however, they do not consider interactions between different levels and the decomposition might not be optimal for video descriptions that focus on actions and events.”
deb2viz: Debiasing gender in word embedding data using subspace visualization (2020, background): “document ranking [1], sentiment analysis [2], computer vision [3]), but propagates biases [4] (e.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2020), which are beneficial for downstream applications, as well as understanding adversarial attacks (Wu et al., 2019) to improve model robustness.”
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022, unknown): “Many propose to learn visual-semantic embedding for vision-text retrieval (Liu et al., 2019; Wu et al., 2019; Lu et al., 2019; Huang et al., 2020; Chen et al., 2021) by attention or objection detection models; and by vision-text contrastive learning (Zhang et al., 2020; Jia et al., 2021; Yuan et…”
Recognize Anything: A Strong Image Tagging Model (2023, unknown): “Initially, we parsed 14 million sentences from our pre-training datasets into tags by utilizing a SceneGraph-Parser [25] with minor modifications.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “including objects, attributes, relationships, and full scenes [117].”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “Hybrid methods have also been used to combine early and late fusion together (Wu et al., 2019; Nguyen & Okatani, 2018; Lazaridou et al., 2015; Hu et al., 2017).”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “Wu et al. (2019) utilizes the structure from semantic parsing in a visual-semantic embedding framework to facilitate embedding learning.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Similar ideas have been also proposed in the image-text alignment by decomposing the images and texts into sub-tokens [26, 50].”