Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark

By Xiaodan Liang, Ke Gong, Xiaohui Shen, Liang Lin. 2018

377 citations recorded by the source corpus.

Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named “Look into Person (LIP)” that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The datasets, code and models are available at http://www.sysu-hcp.net/lip/.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “We conduct experiments to test the effectiveness of visual transformer for semantic segmentation on the COCO-stuff dataset [3] and the LIP dataset [4].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, neutral): “JPPNet [67] Dilated-ResNet-101 Pose 86.”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “81 - JPPNet [39] ResNet-101 8× M - - - 51.”
OCNet: Object Context Network for Scene Parsing (2018, neutral): “e-run all of the experiments for three Table 6: Comparison to state-of-the-art on the validation dataset of LIP. Method Conference Backbone mIoU (%) Attention+SSL [7] CVPR2017 ResNet-101 44:73 JPPNet [11] PAMI2018 ResNet-101 51:37 SS-NAN [36] CVPR2017 ResNet-101 47:92 MMAN [19] ECCV2018 ResNet-101 4”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “We remove the background of these real-world image using neural semantic segmentation [60] followed by Grabcut reﬁnement [61].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “In addition, SSL has been extensively applied to various aspects of image processing and CV: image inpainting [104], human parsing [178], [179], scene deocclusion [180], semantic Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Progressive One-shot Human Parsing (2020, neutral): “…has made signiﬁcant progress by leveraging the large-scale human parsing datasets (Liang et al. 2015b, 2016; Ruan et al. 2019), the parsing results are restricted to the classes pre-deﬁned in the training set, e.g. 18 classes in ATR (Liang et al. 2015a), and 19 classes in LIP (Liang et al. 2018).”
Self-Correction for Human Parsing (2019, methodology): “ion experiments to demonstrate the contribution of each component. 4.1. Datasets and Evaluation Metrics Datasets. We evaluate our proposed method on two single human parsing benchmarks, including LIP [18] and PASCAL-Person-Part [6]. Method hat hair glove s-glass u-clot dress coat sock pant j-suit sc”
Real-Time High-Resolution Background Matting (2020, background): “Specific applications of human segmentation and parsing have also received considerable attention in recent works [33, 19].”
High-Resolution Representations for Labeling Pixels and Regions (2019, neutral): “JPPNet [58] Dilated-ResNet-101 Pose 86.”