Language-driven Semantic Segmentation

By Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, René Ranftl. 2022

928 citations recorded by the source corpus.

We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g.,"grass"or"building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g.,"cat"and"furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, positive): “Note that instead of video-to-text retrieval, but rather on textto-video retrieval, a recent method (Li et al. 2022) has shown strong zero-shot results.”
Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Many open-vocabulary segmentation models [7, 11, 17, 26] leverages large pre-trained vision-language models ( e.g. , CLIP [19] or ALIGN [40]) to distill or transfer the visual-semantic knowledge.”
Generalized Decoding for Pixel, Image, and Language (2022, background): “Prominent examples include LSeg [39], OpenSeg [24], and [31].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “Most prior works on open-vocabulary segmentation either perform object detection with instance segmentation alone [17, 22, 23, 40, 51, 80, 81, 86, 89] or open-vocabulary semantic segmentation alone [22, 36, 76, 88].”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “For instance, LSeg [28] trains a segmentation model as its vision encoder and uses the frozen text encoder from CLIP to align pixel level embeddings with text.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, methodology): “First of all, compared with perpixel approaches (SPNet [37], ZS3Net [4], LSeg [23] and LSeg+ [16]), proposal-based approaches (OpenSeg [16], SimBaseline [40] and ZegFormer [11]) show better performance.”
LangSplat: 3D Language Gaussian Splatting (2023, background): “They learned 3D-consistent features by distilling LSeg [21] or DINO [4] features across multiple views into a NeRF.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “For E 2D , we experiment with two pretrained image segmentation models OpenSeg [17] and LSeg [30].”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, methodology): “Methods such as Distilled Feature Fields [15] and Neural Feature Fusion Fields [43] distill 2D features from pre-trained models such as LSeg [17] and DINO [4] into the radiance ﬁelds, which enable speciﬁcation of regions.”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, methodology): “This allows for more flexibility in the feature learning by constraining only vector directions, and is similarly reflected in CLIP-driven image classification [27,16].”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, background): “VL models possess a strong understanding of open-vocabulary concepts, making them suitable for various downstream vision and vision-language applications [12, 56, 38, 30, 60, 13, 32, 53, 26, 36, 8].”
Transformer-Based Visual Segmentation: A Survey (2023, neutral): “Besidestheseopen vocabularydetectionsettings,recentworks[281],[282]per-formopenvocabularysegmentation.”
Segment Everything Everywhere All at Once (2023, methodology): “Many open-vocabulary segmentation models have been proposed recently [24, 13, 18, 10, 43, 51], which employ large pretrained vision-language models like CLIP [20] to transfer visual-semantic knowledge by freeze or fine-tune their weights.”
Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, methodology): “LSeg [19] aims at dense pixel-wise semantic labeling (without the instance-level formulation) by combining different labeling categories across multiple image segmentation datasets using learned language embedding, but the learned CNN image features in LSeg [19] are not exposed to representations be”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, methodology): “One solution [12,19] to remedy the granularity gap of representation is finetuning the model on the segmentation dataset.”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, background): “There are two concurrent work [30,16] try to utilize the vision-language pre-training model in semantic segmentation.”
Detecting Twenty-thousand Classes using Image-level Supervision (2022, neutral): “ViLD [17], OpenSeg [16] and langSeg [29] upgrade the language embedding to CLIP [42].”