Prototype Mixture Models for Few-shot Semantic Segmentation

By Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, Qixiang Ye. European Conference on Computer Vision, 2020

480 citations recorded by the source corpus.

Few-shot segmentation is challenging because objects within the support and query images could significantly differ in appearance and pose. Using a single prototype acquired directly from the support image to segment the query image causes semantic ambiguity. In this paper, we propose prototype mixture models (PMMs), which correlate diverse image regions with multiple prototypes to enforce the prototype-based semantic representation. Estimated by an Expectation-Maximization algorithm, PMMs incorporate rich channel-wised and spatial semantics from limited support images. Utilized as representations as well as classifiers, PMMs fully leverage the semantics to activate objects in the query image while depressing background regions in a duplex manner. Extensive experiments on Pascal VOC and MS-COCO datasets show that PMMs significantly improve upon state-of-the-arts. Particularly, PMMs improve 5-shot segmentation performance on MS-COCO by up to 5.82\% with only a moderate cost for model size and inference speed.

View Paper: https://www.semanticscholar.org/paper/798e6e0ac92a79c2159a451d900966ffcf0c4ff5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Anti-aliasing Semantic Reconstruction for Few-Shot Semantic Segmentation (2021, methodology): “Under 1-shot settings, with a VGG16 backbone, it respectively outperforms RPMMs [37] and SST [49] by 2.64% and 1.36%.”
Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need? (2020, methodology): “More recently, the dual-branch setting has been unified into a single-branch, employing the same embedding function for both the support and query sets [42, 30, 37, 38, 21].”
Few-Shot Segmentation via Cycle-Consistent Transformer (2021, methodology): “Recent works attempt to generate multiple prototypes via EM algorithm or K-means clustering [36, 16], in order to extract more abundant information from support images.”
Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2022, methodology): “Following the previous works [34, 51, 61], we adopt mean intersection-over-union (mIoU) and foreground-background IoU (FB-IoU) as the evaluation metrics for experiments.”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “An alternative approach might be to use few-shot semantic segmentation [9, 35, 20, 33, 34, 39] to annotate unlabeled images to train image-to-image translation models.”
Image Segmentation Using Text and Image Prompts (2021, background): “Several approaches focus on the modeling of prototypes [37, 38, 39].”
Visual Prompting via Image Inpainting (2022, methodology): “In this setting, the algorithm is trained on a labeled dataset of base classes, from which it should transfer to a set of novel classes given only a few training examples (like 10 or 30) [36, 27, 32, 53, 55, 48, 59, 2].”
CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, background): “Furthermore, several prior works [28, 29, 33, 43] have also claimed that the representation of categories in the feature space should be multiple activations, as shown in Fig.”
Generalized Few-shot Semantic Segmentation (2020, background): “Few-Shot Segmentation (FSSeg) places semantic segmentation in the few-shot scenario [2,8,11,15,18,20,21,24,27,28,33,34,38,41,48,51,52,57, 61, 62, 66, 67], where dense pixel labeling is performed on new classes with only a few support samples.”
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (2021, methodology): “Particularly, ASGNet achieves a margin of 3.98% and 6.96% higher mean IoU over RPMMs [34] in 1-shot and 5-shot segmentation respectively.”
Hypercorrelation Squeeze for Few-Shot Segmenation (2021, methodology): “Following common training/evaluation scheme [37, 46, 70, 74, 80], we conduct cross-validation over all the folds; for each fold i,”