Rethinking Semantic Segmentation: A Prototype View

By Tianfei Zhou, Wenguan Wang, Ender Konukoglu, Luc Van Gool. Computer Vision and Pattern Recognition, 2022

366 citations recorded by the source corpus.

Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manner, our model represents each class as a set of non-learnable prototypes, relying solely on the mean fea-tures of several training pixels within that class. The dense prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space, by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to handle arbitrary number of classes with a constant amount of learnable parameters. We empirically show that, with FCN based and attention based segmentation models (i.e., HR-Net, Swin, SegFormer) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework yields compel-ling results over several datasets (i.e., ADE20K, Cityscapes, COCO-Stuff), and performs well in the large-vocabulary situation. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7475b217b16ce44c64e070f59972e999dca0a771

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, background): “Researchers have significantly promoted image segmentation performance with a variety of deep learning techniques, such as convolutional neural network [34, 35], vision transformers [36, 37], prototype learning [38, 39], etc.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, neutral): “RGB-Depth [38], [39] and RGB-Thermal [40], [41], [42] semantic segmentation are broadly investigated.”