Florence: A New Foundation Model for Computer Vision

By Lu Yuan, Dongdong Chen, Yi-Ling Chen, Yi-ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Chunyuan Li and 13 more. 2021

1,142 citations recorded by the source corpus.

Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “[140] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.”
Visual Instruction Tuning (2023, background): “…language-augmented foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44,…”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[125] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Moreover, our large-size model even outperforms the huge-size model of Florence [48], which also trained on massive image-text pairs using a larger batch size.”
Learning to Prompt for Vision-Language Models (2021, background): “diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “Recent years have seen significant advancement for computer vision, thanks to improved network architecture [4, 9, 12,28], large-scale model training [8,29,35], and other factors.”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “1 AP, even outperforming the giant Florence models [55].”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “In contrast, in the recognition field, the long-standing paradigm has been to build recognition models [32, 42, 84] by starting from a foundation model pretrained on large-scale image data [4,16,17] or image-text pairs [33, 50, 75].”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “We adopt Focal-T [78] and DaViT-B/L [16] as the vision encoder and a transformer text encoder with causal masking [59, 88] as language encoder.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “, 2021), Florence (Yuan et al., 2021), and SimVLM (Wang et al.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The success has quickly inspired many follow-up large-scale pre-training works [68, 72, 69, 36, 43, 20, 34, 74].”
Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “However, we refer the readers to Florence [68] for the large-scale training results, but recommend the suite of experiments in this paper as a baseline for future academic research.”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, methodology): “For example, CoCa [YWV22] uses 65k batch size, CLIP [RKH21] uses 32k batch size, and Florence [YCC21] uses 24k batch size.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “Florence [14] further develops this method with unified contrastive objective [31], training foundation models that can be adapted for a wide range of vision and image-text benchmarks.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “• OFA is pretrained on the publicly available datasets of 20 M image-text pairs, in comparison with recent models that rely on paired data of a much larger scale [22, 23].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “With the introduction of Vision Transformers (Dosovitskiy et al., 2021), vision and language modalities can be jointly modeled by transformers in a more scalable fashion (Yuan et al., 2021; Yu et al., 2022; Wang et al., 2022a; Alayrac et al., 2022).”