PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment
By Kaixin Wang, J. Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng. 2019
1,460 citations recorded by the source corpus.
Despite the great progress made by deep CNNs in image semantic segmentation, they typically require a large number of densely-annotated images for training and are difficult to generalize to unseen object categories. Few-shot segmentation has thus been developed to learn to perform segmentation from only a few annotated examples. In this paper, we tackle the challenging few-shot segmentation problem from a metric learning perspective and present PANet, a novel prototype alignment network to better utilize the information of the support set. Our PANet learns class-specific prototype representations from a few support images within an embedding space and then performs segmentation over the query images through matching each pixel to the learned prototypes. With non-parametric metric learning, PANet offers high-quality prototypes that are representative for each semantic class and meanwhile discriminative for different classes. Moreover, PANet introduces a prototype alignment regularization between support and query. With this, PANet fully exploits knowledge from the support and provides better generalization on few-shot segmentation. Significantly, our model achieves the mIoU score of 48.1% and 55.7% on PASCAL-5i for 1-shot and 5-shot settings respectively, surpassing the state-of-the-art method by 1.8% and 8.6%.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Yi Yang
- Bernt Schiele
- Zeynep Akata
- Jiashi Feng
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Yi Yang
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
In Meta & Continual Learning
- Fabio Cermelli
- Pietro Zanuttigh
- Umberto Michieli
- Barbara Caputo
- Zeynep Akata
- Yongqin Xian
- Alexander S. Ecker
- Timo Lüddecke
Citations in the corpus
- Language-driven Semantic Segmentation (2022, background): “Few-shot approaches (Shaban et al., 2017; Rakelly et al., 2018; Siam et al., 2019; Wang et al., 2019; Zhang et al., 2019; Nguyen & Todorovic, 2019; Liu et al., 2020b; Wang et al., 2020; Tian et al., 2020; Boudiaf et al., 2021; Min et al., 2021) offer ways to learn to segment novel classes based on…”
- Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, methodology): “We use the Frobenius norm || · ||F as metric distance [57, 45, 63].”
- Prototype-based Incremental Few-Shot Segmentation (2020, methodology): “Few-shot Learning approaches can be split in two groups: optimization-based [12, 32, 37, 41] and metric-learning [6, 8, 15, 35, 43, 44, 45, 48, 49].”
- Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, background): “Few-shots learning can be considered as a matching problem where the query is compared with every element in the support set [43, 44, 45, 46].”
- Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation (2022, background): “Prototype-based approaches were initially proposed for few-shot learning [8, 35] and domain adaptation [6, 29, 38].”
- Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need? (2020, methodology): “More recently, the dual-branch setting has been unified into a single-branch, employing the same embedding function for both the support and query sets [42, 30, 37, 38, 21].”
- A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “PANet [202] ICCV’19 Episodic Training Cosine Similarity Matching the similarity of each pixel to the prototype”
- BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation (2021, methodology): “We follow the strategy in recent prototype alignment works [46, 55, 57].”
- Few-Shot Segmentation via Cycle-Consistent Transformer (2021, methodology): “Following previous works [26, 31, 38], both query and support image are first feed into a shared backbone network (e.”
- Progressive One-shot Human Parsing (2020, positive): “Following the one/few-shot learning (Koch, Zemel, and Salakhutdinov 2015; Finn, Abbeel, and Levine 2017; Snell, Swersky, and Zemel 2017; Sung et al. 2018; Chen et al. 2020; Liu et al. 2020a; Tian et al. 2020b), a typical OS3 solution is to learn a good metric (Zhang et al. 2018; Kate et al. 2018; Zh”
- A Few Guidelines for Incremental Few-Shot Segmentation (2020, methodology): “Few-Shot Semantic Segmentation [36, 33, 9, 41, 48, 38] deals with how to learn to segment an image from a scarce training set.”
- Prior Guided Feature Enrichment Network for Few-Shot Segmentation (2020, positive): “The prior masks provide the bias-free prior information from high-level features for both seen and unseen data during the evaluation, while masks produced by learnable feature maps (e.g., [24], [40], [50]) are affected by parameter learning during training.”
- Compositional Prototype Network with Multi-view Comparision for Few-Shot Point Cloud Semantic Segmentation (2020, methodology): “In PANet [3] and CANet [5], support and query branches share the encoder network, and the foreground prototype is computed using global average pooling on the foreground features.”
- Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2022, methodology): “Current FSS methods typically average the prototypes extracted from the support branch and then utilize the averaged features to guide the subsequent segmentation process, which assumes that the contribution of each sample is the same [51, 56].”
- Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (2021, background): “networks [40, 42, 32, 29, 4] merely generate a single prototype by masked average pooling as shown in Figure 1(a), thus losing information as well as discriminability.”
- Prototype Mixture Models for Few-shot Semantic Segmentation (2020, methodology): “-5i. Backbone Method Pascal-50 Pascal-51 Pascal-52 Pascal-53 Mean VGG16 OSLSM [15] 35.90 58.10 42.70 39.10 43.95 SG-One [16] 41.90 58.60 48.60 39.40 47.10 FWB [14] 50.87 62.86 56.48 50.09 55.08 PANet [19] 51.80 64.60 59.80 46.05 55.70 RPMMs (ours) 50.00 66.46 51.94 47.64 54.01 Resnet50 CANet [20] - ”
- Image Segmentation Using Text and Image Prompts (2021, background): “Several approaches focus on the modeling of prototypes [37, 38, 39].”