Unified Perceptual Parsing for Scene Understanding

By Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang. 2018

2,555 citations recorded by the source corpus.

Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes (Models are available at https://github.com/CSAILVision/unifiedparsing).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “We experiment on ADE20K [72] using UperNet [63] (see A.4).”
CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Among the approaches, most of methods [40, 41, 42, 20, 33, 39] adopt the ResNet-101 as backbone and ReﬁneNet [22] adopts a more powerful network, i.e ., ResNet-152, as the backbone.”
Semantic Understanding of Scenes Through the ADE20K Dataset (2016, methodology): “We use UPerNet [35] to jointly train object and part segmentation.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “We use the state-of-the-art segmentation networks for each dataset: DeepLabV2 [5, 40] for COCO-Stuff, UperNet101 [51] for ADE20K, and DRN-D-105 [53] for Cityscapes.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “Our model is built by sequentially stacking networks, which we tried both a convolutional architecture (ResNet-18 backbone [20] and HRNet head [74]) and a transformer architecture (Swin-L backbone [40] and UperNet head [77]).”
K-Net: Towards Unified Image Segmentation (2021, methodology): “B.2 Semantic Segmentation In Table 3b of the main text, we compare K-Net on UperNet [59] using Swin Transformer [39] with the previous state-of-the-art obtained by Swin Transformer [39].”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “Finally, on PASCAL VOC 2012 and ADE20K, our best model outperforms several state-of-the-art models [90, 44, 82, 88, 83] while using strictly less data for pretraining.”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, methodology): “For this purpose we train the Upernet [23] based data-augmented models on the ADE20k dataset with on three different strategies for selecting the object to remove as discussed in Section 3.”
Interpretability Beyond Classification Output: Semantic Bottleneck Networks (2019, methodology): “Broden+ [27] is a recent collection of datasets which serves as a starting point of our case study as it contains annotations for a broad range of relevant semantic concepts.”