Few-Shot Segmentation via Cycle-Consistent Transformer

By Gengwei Zhang, Guoliang Kang, Yunchao Wei, Yi Yang. 2021

271 citations recorded by the source corpus.

Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Cycle-Consistent TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-$5^i$ and COCO-$20^i$ datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art methods by 5.6% and 7.1% respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “Besides, our results are based on a pre-trained ViT backbone for a fair comparison with previous works [46, 45], without considering other representative architectures such as ResNet and other CL applications on downstream computer vision tasks such as object detection [35], image segmentation [54, ”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, unknown): “99 CyCTR [257] NIPS’21 ResNet-50 Metric 64.”
Visual Prompting via Image Inpainting (2022, methodology): “FWB [36] and CyCTR [59] outperform Visual Prompting, mainly because they pretrain on a large tagged base classes dataset and utilize architectures that are speciﬁc to image segmentation.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The multiple images can be support images in few shot segmentation [203], [209], [252] or the same input image in matting [205], [253] and semantic segmentation [206], [207].”