Scene parsing with Multiscale Feature Learning, Purity Trees, and Optimal Covers

By Clément Farabet, Camille Couprie, Laurent Najman, Yann LeCun. 2012

205 citations recorded by the source corpus.

Scene parsing consists in labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained from raw pixels to extract dense feature vectors that encode regions of multiple sizes centered on each pixel. The method alleviates the need for engineered features. In parallel to feature extraction, a tree of segments is computed from a graph of pixel dissimilarities. The feature vectors associated with the segments covered by each node in the tree are aggregated and fed to a classifier which produces an estimate of the distribution of object categories contained in the segment. A subset of tree nodes that cover the image are then selected so as to maximize the average "purity" of the class distributions, hence maximizing the overall likelihood that each segment will contain a single object. The system yields record accuracies on the the Sift Flow Dataset (33 classes) and the Barcelona Dataset (170 classes) and near-record accuracy on the Stanford Background Dataset (8 classes), while being an order of magnitude faster than competing approaches, producing a 320 × 240 image labeling in less than 1 second, including feature extraction.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “The increasing use of these networks for detection in sliding window approaches [9, 28, 33] and the desire to apply CNNs in real-world systems means the speed of inference becomes an important factor for applications.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, methodology): “Prior work on semantic segmentation includes many different approaches, both using RGB-only data [35, 4, 9] as well as RGB-D [31, 29, 26, 6, 15, 17, 13].”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, methodology): “been attempts to apply networks designed for object categorization to segmentation, particularly by replicating the deepest layer features in blocks to match image dimensions [6], [36], [37], [38].”
Fast, Exact and Multi-scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs (2016, methodology): “The currently mainstream approach is relying on ‘Fully’ Convolutional Networks (FCNs) [7,8], where CNNs are trained to provide fields of outputs used for pixelwise labeling.”
DenseNet: Implementing Efficient ConvNet Descriptor Pyramids (2014, neutral): “Fara-bet et al. [16] and Jiu et al. [17]”
Learning Human Pose Estimation Features with Convolutional Networks (2013, background): “Convnets that produce segmentation maps, for example, avoid pooling completely [47, 13].”
Sensor fusion for semantic segmentation of urban scenes (2015, methodology): “Notable exceptions include methods proposed by Farabet et al. [4] and Girshick et al. [5], which learn expressive features from convolutional neural nets.”
Learning Invariant Feature Hierarchies (2012, background): “In recent months, deep learning systems have broken long-held record on a number of benchmarks in a number of areas including speech recognition [22,23,24], object recognition vision [25,26,27], scene parsing [28], action recognition in video [29,30], natural language processing [31,32], and musical”
Machine Learning for Aerial Image Labeling (2013, background): “While image labeling or parsing of general scenes has been extensively studied [He et al., 2004, Shotton et al., 2008, Farabet et al., 2012], aerial images have a few distinct characteristics that make aerial image labeling an easier task.”
Geometric Context from Videos (2013, methodology): “The NYUScenes [6], and MPI-VehicleScenes [19] dataset consists of 74 and 156 annotated frames, respectively.”
Instance Segmentation of Indoor Scenes Using a Coverage Loss (2014, methodology): “Like previous work in semantic segmentation [2] [8] [4] [9], we make use of hierarchical segmentations, referred to as segmentation trees, to limit the search space to a more manageable size.”
Brain-Inspired Classroom Occupancy Monitoring on a Low-Power Mobile Platform (2014, methodology): “Both convolutional neural networks and HMAX have been used to solve a variety of real-world computer vision tasks such as image classification [17][19][21] and scene labeling [9][10].”
Learning Structured Output Representation using Deep Conditional Generative Models (2015, background): “Since the recent success of supervised deep learning on large-scale visual recognition [17, 30, 27], there have been many approaches to tackle mid-level computer vision tasks, such as object detection [6, 26, 31, 9] and semantic segmentation [4, 3, 23, 22], using supervised deep learning techniques.”
Learning Adversary-Resistant Deep Neural Networks (2016, unknown): “Recent explorations and applications include those in medical imaging [1, 38] and self-driving cars [11, 14].”
Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks (2016, unknown): “Recently, multiscale feature learning has been shown to be successful in tasks, such as scene parsing using CNNs [65] and recurrent neural networks [64].”
Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition (2016, background): “In [9], an image is processed at three different scales with three different CNNs, after which the output feature maps are merged.”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling (2015, background): “There have also been a few attempts to apply networks designed for categorization to segmentation, particularly by replicating the deepest layer features in blocks to match image dimensions [7, 6, 11, 8].”