Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

By Yan Zeng, Xinsong Zhang, Hang Li. 2021

379 citations recorded by the source corpus.

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “The perception and external knowledge modules used by ViperGPT are GLIP [31] for object detection, X-VLM [64] for text-image similarity (as it surpasses CLIP [43] at attribute detection [5]), MiDaS [44] for depth estimation, GPT-3 [6] for external knowledge, and BLIP-2 [30] for simple visual…”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “After pretraining, several models exhibit zero-shot capabilities in image-text retrieval (Jia et al., 2021; Radford et al., 2021; Zeng et al., 2022b) and image”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Prominent examples along this line include ViLT [32], ALBEF [36], SimVLM [75], METER [16], X-VLM [85] and BLIP [35].”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Most state-of-the-art VLMs, at their core, are trained with a contrastive loss (Radford et al., 2021; Chen et al., 2020; Zeng et al., 2022a; Li et al., 2021; 2022; Zhang et al., 2020) on a large pretraining dataset scraped from the web.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, methodology): “XVLM is the highest-performing model of those we evaluate, likely due to its more fine-grained supervision at the bounding-box level in addition to the image-level.”
Long-CLIP: Unlocking the Long-Text Capability of CLIP (2024, background): “However, some subsequent works [9,33] have recognized that CLIP lacks the capability of extracting fine-grained information.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, methodology): “Mitigating object hallucination in VLMs has typically involved strategies such as fine-grained contrastive learning [72], ROI feature fusion [6], and the curtailment of co-occurrence patterns via data augmentation [30].”
A survey on multimodal large language models (2023, neutral): “We delineate how LLMs serve these roles in the following parts.”
VLP: A Survey on Vision-language Pre-training (2022, methodology): “3 Text feature extraction For the textual features, following pre-trained language models such as BERT[2], RoBERTa[24], AlBERT[25], and XLNet[26], VLP models[9, 27, 28] first segment the input sentence into a sequence of subwords, and then, insert a start-of-sequence token and an end-of-sequence tok”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, neutral): “6 Acknowledgement”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, background): “…(Su et al., 2019; Chen et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b; Lin et al., 2021; Kim et al., 2021; Dou et al., 2022; Zeng et al., 2021; Li et al., 2021a, 2022), especially in the development of multi-task generalist models (Hu and Singh, 2021; Singh et al., 2022;…”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et al., 2022a,b).”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “While other methods [3, 17, 18, 19] construct multi-level semantic concepts for finer-grained alignment.”
Open-vocabulary Attribute Detection (2022, methodology): “[54] Yan Zeng, Xinsong Zhang, and Hang Li.”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, background): “X-VLM (Zeng et al., 2021) presents an approach that concentrates on multi-grained pretraining.”
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, methodology): “Addressing object hallucination in VLMs is primarily achieved through techniques such as fine-grained contrastive learning (Zeng et al., 2021), ROI feature fusion (Biten et al., 2022), and eliminating co-occurrence patterns through data augmentation (Kim et al., 2023).”