Side Adapter Network for Open-Vocabulary Semantic Segmentation

By Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, Xiang Bai. 2023

461 citations recorded by the source corpus.

This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. Fig. 1 shows some visualization results on ImageNet. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “In the light of strong zero-shot transferability demonstrated in the seminal work CLIP [39], a number of researchers have attempted to build advanced open-vocabulary models by leveraging largescale image-text pairs for fine-grained vision tasks like detection [10, 12, 22, 29, 56] and segmentation [7”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “…the success of PEFT techniques in NLP, a range of PEFT strategies have been proposed and implemented in VLA models, such as semantic segmentation [193], [205], [196], point cloud understanding [219], [241], [181], [81], video understanding [89], [138], [114], visual reasoning [63], [41],…”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “In summary, most approaches [284], [288] adopt the idea of region proposal network [115] to generate class-agnostic mask proposals via different approaches, including anchor-based and query-based decoders in Section III-A.”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Therefore, feature adaptation has clear advantages in adapting VLMs to work on very different and complex downstream tasks [168], [169], [170], [171].”
Segment Everything Everywhere All at Once (2023, background): “Many recent works proposed to either leverage contrastive learning methods or pretrained multi-modal foundation models ( e.g. , CLIP [8]) to make the segmentation models more transferable to unseen concepts [9, 10, 11, 12]; 2) From generic to referring segmentation.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, unknown): “Pre-trained CLIP has em-powered many computer vision tasks with the language ability, such as image manipulation [34], image generation [10], object detection [17, 42] and image segmentation [11, 12, 16, 21, 23, 31, 39, 40].”