Learning to place new objects in a scene

By Yun Jiang, Marcus Lim, Changxi Zheng, Ashutosh Saxena. 2012

159 citations recorded by the source corpus.

Placing is a necessary skill for a personal robot to have in order to perform tasks such as arranging objects in a disorganized room. The object placements should not only be stable but also be in their semantically preferred placing areas and orientations. This is challenging because an environment can have a large variety of objects and placing areas that may not have been seen by the robot before. In this paper, we propose a learning approach for placing multiple objects in different placing areas in a scene. Given point-clouds of the objects and the scene, we design appropriate features and use a graphical model to encode various properties, such as the stacking of objects, stability, object–area relationship and common placing constraints. The inference in our model is an integer linear program, which we solve efficiently via an linear programming relaxation. We extensively evaluate our approach on 98 objects from 16 categories being placed into 40 areas. Our robotic experiments show a success rate of 98% in placing known objects and 82% in placing new objects stably. We use our method on our robots for performing tasks such as loading several dish-racks, a bookshelf and a fridge with multiple items.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, neutral): “Our work complements previous works on grasping items [30, 21], pick and place tasks [11], and detecting bar code for grocery checkout [16].”
Learning human activities and object affordances from RGB-D videos (2012, methodology): “In recent work, Jiang et al. (2012a) used a data-driven technique for learning spatial affordance maps for objects.”
Actions ~ Transformations (2015, background): “More recently, trajectory based approaches [51, 31, 29, 50, 18, 32, 25] have shown signiﬁcant improvement in action recognition.”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, methodology): “[14] propose a method to place a new object in a stable and semantically preferred location in a scene.”
Unstructured human activity detection from RGBD images (2011, methodology): “We have also made the dataset and code available open-source at: http://pr.cs.cornell.edu/humanactivities”
Semantically-enriched 3D models for common-sense knowledge (2015, neutral): “Which way does the fridge open?”
Metric learning for generalizing spatial relations to new objects (2017, methodology): “Other methods have also relied on the geometries of objects and scenes to reason about preferred object placements [11] or likely places to ﬁnd an object [3].”
Contextually Guided Semantic Labeling and Search for 3D Point Clouds (2011, unknown): “(For more details on the hardware speciﬁcation of the POLAR robot, please see Jiang et al., 2012.)”
Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning (2017, background): “Learning of object interactions from contact distributions has been addressed in the context of object grasping [12] and object placing [13].”
Learning preferences for manipulation tasks from online coactive feedback (2015, result): “Our work complements previous works on grasping items [49, 34], pick and place tasks [23], and detecting bar codes for grocery checkout [29].”
Contextually guided semantic labeling and search for three-dimensional point clouds (2013, methodology): “(For more details on the hardware specification of the POLAR robot, please see Jiang et al., 2012.)”
Learning hierarchical sparse features for RGB-(D) object recognition (2014, background): “Jiang et al. (2012) proposed a supervised learning approach for detecting good placements from the given point clouds of an object and the placing area.”
A data-driven statistical framework for post-grasp manipulation (2014, background): “[11] looked at scenes to determine good locations to place objects.”
Learning Object Arrangements in 3D Scenes using Human Context (2012, methodology): “Learning such aﬀordances for reasoning about the objects’ arrangements would be useful in several areas of scene understanding (e.g., Koppula et al., 2011) or assistive robots (e.g., Jiang et al., 2012a).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Representing and Synthesizing Skill Parameters: Similar to the predicate representations, these policy parameter arguments can be modeled as additional input features for the precondition classifier or as distributions over valid and invalid argument values [86, 180].”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Reasoning from Blocks to Stability (background): “Xiao et al. [13] detect 3D cuboids with arbitrary orientations solely in RGB images, Bleyer et al. [14] show box fitting for improved stereo, and Jiang et al. [15] propose a linear programming for fitting cuboids in depth images.”