Look into Person: Self-Supervised Structure-Sensitive Learning and a New Benchmark for Human Parsing

By Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, Liang Lin. 2017

511 citations recorded by the source corpus.

Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark Look into Person (LIP) that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analysis of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Human part segmentation masks have been provided in a number of datasets [48, 6, 13]; these can be understood as providing a coarsened version of image-to-surface correspondence, where rather than continuous coordinates one predicts discretized part labels [35].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “The LIP dataset [47] contains 50 , 462 elaborately annotated human images, which are divided into 30 , 462 training images, and 10 , 000 validation images.”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “The LIP dataset [18] is used in the LIP challenge 2016 for single human parsing task.”
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting (2019, background): “mainly follow two lines: i) transform annotations from other domain to object masks [12,30] or ii) utilize data from other domain as extra regularization term [21,4].”
Human Semantic Parsing for Person Re-identification (2018, methodology): “Despite being out of the scope of this work, to demonstrate the quality of our human semantic parsing, we show in Table 1 that, on the validation set of LIP [14], our model outperforms the current state-of-the-”
Progressive One-shot Human Parsing (2020, unknown): “Recent methods for human parsing improve parsing performance from utilizing the body structure priors and class relations (Xiao et al. 2018; Gong et al. 2017; Zhu et al. 2018; Gong et al. 2018; Li et al. 2020; Zhan et al. 2019).”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, methodology): “It provides high quality pixel-level annotations for 5 , 000 frames and coarse annotations for 20 , 000 frames. dataset is annotated with 30 street-specific classes, such as car , road , traffic sign , etc. Finally, the LIP dataset is focused on the semantic segmentation of people and provides more ”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, background): “[17] introduced a generative model that can generate human parsing [8] maps and translate them into persons in clothing.”
OCNet: Object Context for Semantic Segmentation (2021, background): “We compare our approach to the previous state-of-the-arts on LIP val and illustrate the results in Table 10.”
Knowledge Integration Networks for Action Recognition (2020, methodology): “LIP is a human parsing dataset, which was created speciﬁcally for semantic segmentation of multiple parts of human body.”
Self Adversarial Training for Human Pose Estimation (2017, methodology): “Second, we evaluate our method using LSP, MPII, and LIP datasets, with detailed analysis on the effects of different components in our design, and the experimental results show improved accuracy on all of those datasets.”
Feature Pyramid Transformer (2020, methodology): “0% mIoU on ADE20K [24] and LIP [25] validation sets, respectively.”
Perceive Where to Focus: Learning Visibility-Aware Part-Level Features for Partial Person Re-Identification (2019, background): “[23] extract several region parts, with pose estimation [17, 27, 10, 18, 1], human parsing [2, 5] and uniform partitioning, respectively.”
Instance-level Human Parsing via Part Grouping Network (2018, methodology): “[11] designed a structure-sensitive learning to enforce the produced parsing results semantically consistent with the human joint structures.”
High-Resolution Representations for Labeling Pixels and Regions (2019, methodology): “ic segmentation is a problem of assigning a class label to each pixel. We report the results over two scene parsing datasets, PASCAL Context [71] and Cityscapes [19], and a human parsing dataset, LIP [34]. The mean of class-wise intersection over union (mIoU) is adopted as the evaluation metric. Cit”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “In addition, SSL has been extensively applied to various aspects of image processing and CV: image inpainting [104], human parsing [178], [179], scene deocclusion [180], semantic Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
VITON: An Image-Based Virtual Try-on Network (2017, methodology): “A state-of-the-art human parser [11] is thus used to compute a human segmentation map, where different regions represent different parts of human body like arms, legs, etc.”