FLAVA: A Foundational Language And Vision Alignment Model

By Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela. 2021

987 citations recorded by the source corpus.

State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a “foundation”, that targets all modalities at once-a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

Subfields: Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[101] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.”
Learning to Prompt for Vision-Language Models (2021, background): “diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “[68] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, background): “However, recently several “generalist” models have been developed, such as UnifiedIO [Lu et al., 2022] and FLAVA [Singh et al., 2022], that can solve a large variety of multimodal and unimodal tasks by formulating all tasks as a Sequence-to-Sequence problem.”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, neutral): “Model Flickr30K (1K test set) Image→ Text Text→ Image R@1 R@5 R@10 R@1 R@5 R@10 FLAVA [SHG21] 67.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “Unlike other fusion-based foundation methods [16, 17, 35], CoCa is naturally applicable to crossmodal alignment tasks since it generates aligned image and text unimodal embeddings.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “[59] and [60] unify tasks of different modalities by designing various task-speciﬁc layers.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “There has been a considerable body of research on learning representations of multiple modalities simultaneously often using paired data (Aytar et al., 2017; Radford et al., 2021b; Wang et al., 2021; Singh et al., 2021) with the aim to produce cross-modal representations which can perform well on mu”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “For example, the Public Multimodal Dataset (PMD) [92] and LAION-Aesthetics [85] datasets consist almost entirely of images drawn from LAION-2B-en.”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, background): “Objectives include contrastive and discriminative image–text matching [40,53,69] including local variants [32,76], auto-regressive (AR) captioning [4, 39, 77] and multi-modal masked modelling objectives [13, 40, 61].”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “We train using a union of several datasets: CC12m (Changpinyo et al., 2021), CC (Sharma et al., 2018), YFCC100m (ﬁltered) (Thomee et al., 2016; Singh et al., 2022), Redcaps (Desai et al., 2021), LAION-aesthetic-6+ (Schuhmann et al., 2022).”
Visual Classification via Description from Large Language Models (2022, methodology): “Other work such as ALIGN (Jia et al., 2021), FLAVA (Singh et al., 2022), Florence (Yuan et al., 2021), and more have since furthered this paradigm.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, background): “GPV [18] proposes a general-purpose vision system, and FLAVA [58] presents a VL system similar to METER [13]; however, they did not evaluate on grounding and detection tasks, and their performance on other VL tasks is still far from SoTA.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, methodology): “We evaluate various conﬁgurations of the following multimodal transformers: CLIP [56], FLAVA [68], LXMERT [76], UniT [35], UNITER [12], VILLA [23], VinVL [90], ViLT [40], VisualBERT [47] and ViLBERT [51].”
A Cookbook of Self-Supervised Learning (2023, background): “Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “…generative and contrastive objectives (Yu et al., 2022); XVLM (Zeng et al., 2022) with 4M and 16M parameters; BLIP (Li et al., 2022) with 14M and 129M parameters; BLIP2 (Li et al., 2023) image-text matching head (ITM) and image-text contrastive learning head (ITC); and FLAVA (Singh et al., 2022).”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, background): “In particular, one of the seemingly likely candidates of multi-modal fusion for generating patch level alignment could be cross-attention between image and text tokens, often seen in architectures used in VLP training [25, 31, 45] etc.”