Prior Guided Feature Enrichment Network for Few-Shot Segmentation

By Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, Jiaya Jia. 2020

723 citations recorded by the source corpus.

State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results and hardly work on unseen classes without fine-tuning. Few-shot segmentation is thus proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information of training classes and spatial inconsistency between query and support targets. To alleviate these issues, we propose the Prior Guided Feature Enrichment Network (PFENet). It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks. Extensive experiments on PASCAL-5<inline-formula><tex-math notation="LaTeX">$^i$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mi>i</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="tian-ieq1-3013717.gif"/></alternatives></inline-formula> and COCO prove that the proposed prior generation method and FEM both improve the baseline method significantly. Our PFENet also outperforms state-of-the-art methods by a large margin without efficiency loss. It is surprising that our model even generalizes to cases without labeled support samples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Language-driven Semantic Segmentation (2022, background): “…et al., 2017; Rakelly et al., 2018; Siam et al., 2019; Wang et al., 2019; Zhang et al., 2019; Nguyen & Todorovic, 2019; Liu et al., 2020b; Wang et al., 2020; Tian et al., 2020; Boudiaf et al., 2021; Min et al., 2021) offer ways to learn to segment novel classes based on only a few labeled images.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “For example, few-shot learning [21, 43, 50, 55, 70, 77, 84] and active learning [8, 62, 66, 67, 83] are proposed to perform segmentation with as few pixel-wise labels as possible.”
Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need? (2020, methodology): “Particularly, we trained the methods in [33, 38] by using one labeled sample per class, i.”
RoMa: Robust Dense Feature Matching (2023, positive): “A well-known approach to limit overfitting is to freeze the backbone used [29, 47, 54].”
Hypercorrelation Squeeze for Few-Shot Segmenation (2021, methodology): “8%p of mIoU improvements over [70] and [4], verifying its superiority in few-shot segmentation task.”
Visual Prompting via Image Inpainting (2022, methodology): “In this setting, the algorithm is trained on a labeled dataset of base classes, from which it should transfer to a set of novel classes given only a few training examples (like 10 or 30) [36, 27, 32, 53, 55, 48, 59, 2].”
Image Segmentation Using Text and Image Prompts (2021, methodology): “PFENet [40] uses a prior computed on high-level CNN-features to provide an auxiliary segmentation that helps further processing.”
Progressive One-shot Human Parsing (2020, neutral): “…2015; Finn, Abbeel, and Levine 2017; Snell, Swersky, and Zemel 2017; Sung et al. 2018; Chen et al. 2020; Liu et al. 2020a; Tian et al. 2020b), a typical OS3 solution is to learn a good metric (Zhang et al. 2018; Kate et al. 2018; Zhang et al. 2019b,a; Hu et al. 2019; Tian et al. 2020a).”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, neutral): “3 SVF [260] Arxiv’22 ResNet-50 Transfer learning 60.”
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (2021, methodology): “networks [40, 42, 32, 29, 4] merely generate a single prototype by masked average pooling as shown in Figure 1(a), thus losing information as well as discriminability.”
Generalized Few-shot Semantic Segmentation (2020, methodology): “Besides, we also incorporate CAPL to PFENet whose decoder processes the concatenation of the prior mask and middle-level features to make predictions.”
Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2022, methodology): “Current FSS methods typically average the prototypes extracted from the support branch and then utilize the averaged features to guide the subsequent segmentation process, which assumes that the contribution of each sample is the same [51, 56].”
Few-Shot Segmentation via Cycle-Consistent Transformer (2021, methodology): “PFENet [26] further designs an effective feature pyramid module and leverages a prior map to achieve better segmentation performance.”