Learning Object Arrangements in 3D Scenes using Human Context

By Yun Jiang, Marcus Lim, Ashutosh Saxena. 2012

113 citations recorded by the source corpus.

We consider the problem of learning object arrangements in a 3D scene. The key idea here is to learn how objects relate to human poses based on their affordances, ease of use and reachability. In contrast to modeling object-object relationships, modeling human-object relationships scales linearly in the number of objects. We design appropriate density functions based on 3D spatial features to capture this. We learn the distribution of human poses in a scene using a variant of the Dirichlet process mixture model that allows sharing of the density function parameters across the same object types. Then we can reason about arrangements of the objects in the room based on these meaningful human poses. In our extensive experiments on 20 different rooms with a total of 47 objects, our algorithm predicted correct placements with an average error of 1:6 meters from ground truth. In arranging five real scenes, it received a score of 4.3/5 compared to 3.7 for the best baseline method.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, background): “In previous work, such relations were considered in the context of scene understanding [10, 12].”
Learning human activities and object affordances from RGB-D videos (2012, methodology): “Recent works in this area have addressed tasks such as estimating geometry (Henry et al., 2012), tracking objects (Choi and Christensen, 2012), recognizing objects (Collet et al., 2011), placing objects (Jiang et al., 2012b) and labeling geometric classes (Koppula et al., 2011; Anand et al., 2012).”
Anticipating Human Activities Using Object Affordances for Reactive Robotic Response (2013, background): “Some works use human pose for action recognition by detecting local pose features [25] and modeling the spatial configuration between human body parts and objects [5], [26], [27], [28], [29].”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, methodology): “proposed methods that use human context to learn object arrangement [287] and object labeling [110].”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, background): “Another direction models human-scene interactions by hallucinating synthetic people either in real RGB images of scenes [29] for general scene labeling, or in synthetic 3D scenes to learn affordances [21, 33] or 3D object layout in the scene [30], or in real 3D scans of scenes [16] for scene synthes”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Reasoning from Blocks to Stability (background): “Further, the 3D structure of indoor scenes has been studied through affordances, as in [21] [22] and [23].”
PiGraphs (2016, background): “Another line of work encodes human interactions from RGB-D data to hallucinate plausible human poses for labeling objects or for learning priors on the spatial distribution of objects with respect to people [Jiang et al. 2012; Jiang et al. 2013; Jiang and Saxena 2013].”
Hallucinating Humans for Learning Robotic Placement of Objects (2012, unknown): “Our dataset contains 20 scenes (7 kitchens, 6 living rooms and 7 offices) and 47 objects from 19 categories that are commonly seen in these scenes [13].”
Hallucinated Humans as the Hidden Context for Labeling 3D Scenes (2013, background): “We therefore define F as (see [14]): F (xi; θ H , θ) = FdistFrelFheight, (2)”
Learning task-oriented grasping for tool manipulation from simulated self-supervision (2018, methodology): “Prior art has developed methods to learn different forms of object affordance such as semantic labels [56], spatial maps [24], and motion trajectories [55].”
SceneGrok (2014, background): “Another line of work hypothesizes human poses as a latent variable and predicts plausible poses in scenes in order to label objects, or determine likely placements of objects [Jiang et al. 2012; Jiang et al. 2013; Jiang and Saxena 2013].”
Patch to the Future: Unsupervised Visual Prediction (2014, background): “Recently, some researchers have started focusing on modeling static functional interactions of humans with the scene [11, 15] and objects [9, 31] which can be then be used to predict how will humans interact with the scene.”