Panoptic Studio: A Massively Multiview System for Social Interaction Capture

By Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart C. Nabbe, Iain Matthews and 3 more. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016

459 citations recorded by the source corpus.

We present an approach to capture the 3D motion of a group of people engaged in a social interaction. The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion needs to be measured over a space large enough to host a social group; (3) human appearance and configuration variation is immense; and (4) attaching markers to the body may prime the nature of interactions. The Panoptic Studio is a system organized around the thesis that social interactions should be measured through the integration of perceptual analyses over a large variety of view points. We present a modularized system designed around this principle, consisting of integrated structural, hardware, and software innovations. The system takes, as input, 480 synchronized video streams of multiple people engaged in social activities, and produces, as output, the labeled time-varying 3D structure of anatomical landmarks on individuals in the space. Our algorithm is designed to fuse the “weak” perceptual processes in the large number of views by progressively generating skeletal proposals from low-level appearance cues, and a framework for temporal refinement is also presented by associating body parts to reconstructed dense 3D trajectory stream. Our system and method are the first in reconstructing full body motion of more than five people engaged in social interactions without using markers. We also empirically demonstrate the impact of the number of views in achieving this goal.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/bcb1d5f285cd2a3c00fbdb82a65fca1bed1e9f28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Deep 3D human pose estimation: A review (2021, unknown): “CMU Panoptic Joo et al. (2017) provides some examples with large social interaction.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “HOnnotate [32] relies on an optimization process to estimate accurate hand and object pose from Dataset Frames Action 6D Obj 3D left 3D right (**)Markerless Real Ego Depth Multiview H2O 571k ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ FPHA [26] 100k ✓ * (23k) · ✓ · ✓ ✓ ✓ · HOnnotate [32] 78k · ✓ · ✓ ✓ ✓ · ✓ ✓ Obman [34] 150k”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “Datasets play an important role in deep learning-baed human pose estimation.”
Learnable Triangulation of Human Pose (2019, methodology): “6M [3] and CMU Panoptic [5, 20, 15].”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “, 2018), 3D keypoints (Ionescu et al., 2013; Joo et al., 2015; Mehta et al., 2017; Trumble et al., 2017; Li et al., 2021) and statistical model parameters (Yu et al.”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, unknown): “MTC. Monocular Total Capture [55] is a dataset captured by Panoptic Studio [16, 18] in a multi-view setup with 30 HD cameras.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “This variety means Ego-Exo4D augments existing 3D human body pose datasets [49, 66, 68, 80, 193].”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Readers are referred to the original references for details about other datasets including MPI-INF-3DHP [157], HumanEva [217], CMU Panoptic Dataset [92], TotalCap-ture [234], MuCo-3DHP Dataset [159], 3DPW [240], AMASS [152], NBA2K [314], GTA-IM [15], and Occlusion-Person [295].”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “(ii) The dataset proposed and collected by [19], which also contains 3D poses of multiple persons in social interactions.”