Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

By Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, Lu Yuan. Computer Vision and Pattern Recognition, 2023

565 citations recorded by the source corpus.

We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for various computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform diverse tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with un-precedented zero-shot and fine-tuning capabilities.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/441bada9aa6dfd1f94d45d20e0f7eb060d59dd30

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Step1X-Edit: A Practical Framework for General Image Editing (2025, unknown): “For subject-add and subject-remove tasks, we begin by annotating our proprietary dataset using Florence-2 [60], which supports diverse semantic granularities, spatial hierarchies, and annotation types such as object detection and classification.”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, methodology): “…phase using a diverse dataset, which consists of a combination of interleaved image-text documents ( e.g. , [LST + 24]), image-text pairs from FLD-5B [XWX + 24], synthetic data derived from Optical Character Recognition (OCR) of PDF files, datasets for chart/table comprehension, and text-only data.”
ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “The most related model is a 176M late interaction model, ColFlor (Masry & Hoque, 2024), trained from the Florence2 model (Xiao et al., 2023).”
Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI (2025, unknown): “TinyClick and UGround are fine-tuned versions of Florence-2-Base [34] and Qwen2-VL [30], respectively.”