Contextually Guided Semantic Labeling and Search for 3D Point Clouds

By Abhishek Anand, Hema Swetha Koppula, Thorsten Joachims, Ashutosh Saxena. 2011

79 citations recorded by the source corpus.

RGB-D cameras, which give an RGB image to- gether with depths, are becoming increasingly popular for robotic perception. In this paper, we address the task of detecting commonly found objects in the 3D point cloud of indoor scenes obtained from such cameras. Our method uses a graphical model that captures various features and contextual relations, including the local visual appearance and shape cues, object co-occurence relationships and geometric relationships. With a large number of object classes and relations, the model's parsimony becomes important and we address that by using multiple types of edge potentials. We train the model using a maximum-margin learning approach. In our experiments over a total of 52 3D scenes of homes and offices (composed from about 550 views), we get a performance of 84.06% and 73.38% in labeling office and home scenes respectively for 17 object classes each. We also present a method for a robot to search for an object using the learned model and the contextual information available from the current labelings of the scene. We applied this algorithm successfully on a mobile robot for the task of finding 12 object classes in 10 different offices and achieved a precision of 97.56% with 78.43% recall.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning human activities and object affordances from RGB-D videos (2012, background): “…by modeling objects and their relations to other objects in the scene (Koppula et al., 2011; Lai et al., 2011b; Rosman and Ramamoorthy, 2011; Anand et al., 2012), developing frameworks for object recognition and pose estimation for manipulation (Collet et al., 2011), object tracking for 3D…”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, unknown): “Given a 3D scene as input, 3D semantic segmentation task aims to assign a semantic category to each point in the scene [2–4, 9, 14, 15, 22, 28, 29, 31, 38, 42, 44, 46, 47, 53, 54, 63, 64, 66, 68, 70, 73].”
RAPter (2015, methodology): “2014], or aid autonomous navigation by providing scene maps [Anand et al. 2011].”
3D point cloud segmentation: A survey (2013, background): “Recently, Anand [2] used contextual relations such as local visual appearance, shape cues, and geometric relationships combine with graphical model to semantic segment point clouds of indoor scenes and achieve good results.”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, background): “find(obj) Find obj in the environment [2].”
Detecting potential falling objects by inferring human action and natural disturbance (2014, background): “The recent development of consumer-grade range cameras, such as the Kinect camera, has attracted increasing studies in the field of 3D scene understanding [4] [9] [13] [21].”
Dense 3D semantic mapping of indoor scenes from RGB-D images (2014, background): “Several 3D semantic segmentation approaches exist for both outdoor [11], [12], [13] and indoor [3], [14], [2], [8] scenes.”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Reasoning from Blocks to Stability (background): “…that integrating depth with color information improves the performances of many vision tasks, such as segmentation (in [7]), contour detection (in [24]), object recognition (in [25], [26], and [27]), scene labeling (in [28], [29], [30] and [31]), and activity detection (in [32], [33] and [34]).”
Learning hierarchical sparse features for RGB-(D) object recognition (2014, background): “Anand et al. (2012) detect commonly found objects in the 3D point cloud of indoor scenes obtained from RGB-D cameras.”
SEGCloud: Semantic Segmentation of 3D Point Clouds (2017, background): “…in 3D space, which is useful for ﬁne-grained tasks such as object manipulation, detailed scene modeling, etc. Semantic segmentation of 3D point sets or point clouds has been addressed through a variety of methods leveraging the representational power of graphical models [36, 44, 3, 48, 30, 35].”
Hallucinated Humans as the Hidden Context for Labeling 3D Scenes (2013, methodology): “We used the Cornell RGB-D indoor dataset [19, 2] for our experiments.”