Indoor Semantic Segmentation using depth information

By Camille Couprie, Clément Farabet, Laurent Najman, Yann LeCun. 2013

512 citations recorded by the source corpus.

This work addresses multi-class segmentation of indoor scenes with RGB-D inputs. While this area of research has gained much attention recently, most works still rely on hand-crafted features. In contrast, we apply a multiscale convolutional network to learn features directly from the images and the depth information. We obtain state-of-the-art on the NYU-v2 depth dataset with an accuracy of 64.5%. We illustrate the labeling of indoor scenes in videos sequences that could be processed in real-time using appropriate hardware such as an FPGA.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In 3D Vision & Pose

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “We evaluate performance on the 13-class [18] and 40-class [28] segmentation tasks, and we report mean intersection-over-union (mIoU) as our evaluation metric.”
Learning Rich Features from RGB-D Images for Object Detection and Segmentation (2014, methodology): “[6] adapt the multiscale semantic segmentation system of Farabet et al.”
Stacked Hourglass Networks for Human Pose Estimation (2016, background): “Our work is closely connected to fully convolutional networks [23] and other designs that process spatial information in multiple scales for dense prediction [15,30,31,32,33,34,35,36,37,38].”
Recent advances in convolutional neural networks (2015, methodology): “Besides, this network is also successfully applied to RGB-D scene labeling [186].”
Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding (2015, background): “In particular deep learning [25] has set the benchmark on many popular datasets [11, 8].”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, background): “Feature learning for 3D recognition has ﬁrst appeared in the context of RGB-D images, where depth is treated as an additional input channel [1, 6, 8].”
Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks (2016, background): “Recently, CNNs have been shown to be successful in object recognition [17,18], object detection [19], scene parsing [20,21] and scene classification [22–24].”
Deep Multimodal Data Fusion (2024, methodology): “For example, for the image segmentation task, Couprie et al. [27] propose the first deep learning-based multimodal fusion model.”
Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community (2017, methodology): “Couprie et al.271 used a multiscale CNN to learn features directly from RGBD imagery.”
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation (2017, methodology): “In contrast, recent methods [5, 11, 7, 25, 39, 13] usually employ DCNN that automatically trains features capturing different levels of representations.”
Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition (2015, methodology): “[11] presented a multi-scale CNN framework for RGB-D”
Survey on semantic segmentation using deep learning techniques (2019, methodology): “In deep networks for semantic segmentation, CNNs are applied to image square patches, often called kernel of ﬁxed size Couprie et al. [ 109] adopted a similar approach, and proposed a convolutional network to learn multi-scale features us-ing image depth information.”
Understanding RealWorld Indoor Scenes with Synthetic Data (2015, methodology): “has been used in [8, 22, 10] on per-pixel scene understanding.”
LLNet: A deep autoencoder approach to natural low-light image enhancement (2015, background): “…approaches gained immense traction as they are shown to outperform other state-of-the-art machine learning tools in many computer vision applications, including object recognition (Krizhevsky et al., 2012), scene understanding (Couprie et al., 2013) and occlusion detection (Sarkar et al., 2015).”