FILIP: Fine-grained Interactive Language-Image Pre-Training

By Lewei Yao, Runhu Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, Chunjing Xu. 2021

865 citations recorded by the source corpus.

Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[138] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Table 1 shows the results over these 12 datasets, in comparison with the best performance achieved by both CLIP ResNet and Vision Transformer models, and the concurrent work FILIP (Yao et al., 2021).”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…also been proposed over the years, and have progressively converged to a few time-tested ones: image-text contrastive learning (Radford et al., 2021; Yao et al., 2022; Li et al., 2021; 2022), image-text matching (Li et al., 2021; 2022; Wang et al., 2021a), and (masked) language modeling (Li et…”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The success has quickly inspired many follow-up large-scale pre-training works [68, 72, 69, 36, 43, 20, 34, 74].”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “FILIP Yao et al. (2021) leverages the finer-grained alignment between image patches and textual words.”
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks (2023, background): “Pretraining Tasks Many multimodal pretraining objectives have been proposed, including image-text contrastive learning [24, 45, 68], image-text matching [26, 31, 56, 62], masked language modeling [17, 26, 34, 54, 56] or prefix language modeling [63], masked region classification [56], word-patch/reg”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “CC3M [47], CC12M [4], YFCC15M [43, 51]), by exploring pretraining on a unified image-text-label space [59], token-level contrastive loss [61], and auxiliary within-modality contrastive loss [29,37,58,63].”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “FILIP [110] focuses on achieving ﬁner-level alignment in multimodal learning.”
Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “Recently, CLIP [69], ALIGN [37] and FILIP [90] have further scaled up the training with large-scale web data.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “Another line of research uses a dual-stream architecture (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2021; Yao et al., 2021) with separate image and text encoders aligned together through an image-text contrastive loss.”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “FILIP [8] improves the contrastive objective to achieve finer-level alignment, by using a token-wise maximum similarity between visual and textual tokens.”
Unsupervised Prompt Learning for Vision-Language Models (2022, methodology): “Recently, vision-language models such as CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021) and FLIP (Yao et al. 2021) have achieved promising progress in visual representation learning and transfer learning.”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, background): “Vision Language models: Foundational vision-language (VL) models [35, 20, 54, 49, 51] leverage both visual and textual modalities to encode rich multi-modal representations.”
Disentangled Representation Learning for Text-Video Retrieval (2022, methodology): “Token-wise Interaction: Recently, ColBERT [18] and FILIP [45] propose token-wise interaction for document and image retrieval.”