PiGraphs

By Manolis Savva, Angel X. Chang, P. Hanrahan, Matthew Fisher, Matthias Nießner. 2016

118 citations recorded by the source corpus.

We learn a probabilistic model connecting human poses and arrangements of object geometry from real-world observations of interactions collected with commodity RGB-D sensors. This model is encoded as a set of prototypical interaction graphs (PiGraphs), a human-centric representation capturing physical contact and visual attention linkages between 3D geometry and human body parts. We use this encoding of the joint probability distribution over pose and geometry during everyday interactions to generate interaction snapshots, which are static depictions of human poses and relevant objects during human-object interactions. We demonstrate that our model enables a novel human-centric understanding of 3D content and allows for jointly generating 3D scenes and interaction poses given terse high-level specifications, natural language, or reconstructed real-world scene constraints.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, methodology): “The semantic seg-mentations are more than an order of magnitude larger than any previous RGB-D dataset.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “Existing work on curation of RGB-D datasets has focused mostly on scans of individual objects [5], stand-alone rooms [20, 28], views of a room [32, 35], spaces from academic buildings and small apartments [7], and small collections of rooms or public spaces [44, 1, 22].”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, methodology): “Future Work: GRAB can support learning human-object interaction models [42, 61], robotic grasping from imitation [74], learning mappings from MoCap markers to meshes [20], rendering synthetic images [24, 53, 70], inferring object shape/pose from interaction [2, 44], or analysis of temporal patterns ”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, methodology): “Neither our PROX dataset nor PiGraphs [62] have ground-truth for quantitative evaluation.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “While recent work has also demonstrated impressive results in estimating 3D structures, particularly human bodies, the focus is often on bodies [30,37] and objects [11,17] imaged in isolation or in controlled lab conditions [23,56].”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, background): “Scene-centric datasets can be broadly divided into indoor scens [5, 10, 18, 39, 58, 63, 75, 81, 84, 104], and outdoor (self-driving focussed) scenes [26, 71, 88].”
ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data (2021, background): “There are several other datasets which have focused on long indoor RGB-D captures [9, 14, 23, 24] to address the view-point diversity.”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “tivities [11,39,50]. However, such coarse-grained multi-agent interactions leave the latent subtlety of collaboration and task assignment untouched. Although simulation-based multi-agent environments [3,59,6] can partially address such an issue, learning from noisy and real visual input in physical ”