Nonparametric scene parsing: Label transfer via dense scene alignment

By Ce Liu, Jenny Yuen, Antonio Torralba. 2009

364 citations recorded by the source corpus.

In this paper we propose a novel nonparametric approach for object recognition and scene parsing using dense scene alignment. Given an input image, we retrieve its best matches from a large database with annotated images using our modified, coarse-to-fine SIFT flow algorithm that aligns the structures within two images. Based on the dense scene correspondence obtained from the SIFT flow, our system warps the existing annotations, and integrates multiple cues in a Markov random field framework to segment and recognize the query image. Promising experimental results have been achieved by our nonparametric scene parsing system on a challenging database. Compared to existing object recognition approaches that require training for each object category, our system is easy to implement, has few parameters, and embeds contextual information naturally in the retrieval/alignment procedure.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Ensemble of exemplar-SVMs for object detection and beyond (2011, background): “Because most detections show good alignment to their associated exemplar, it is possible to transfer any available exemplar meta-data (segmentation, geometric structure, 3D model, etc.) directly onto the detections, which can then be used as part of overall scene understanding.”
Recurrent Convolutional Neural Networks for Scene Parsing (2013, methodology): “We tested our proposed method on two diﬀerent datasets for scene parsing: the Stanford Background [1] and the SIFT Flow Dataset [5].”
Image Segmentation Using Deep Learning: A Survey (2020, background): “SiftFlow [155] includes 2,688 annotated images, from a subset of the LabelMe database, of 8 different outdoor scenes, among them streets, mountains, fields, beaches, and”
Multi-Atlas Segmentation with Joint Label Fusion (2012, background): “Recently, the concept has also been applied in computer vision for segmenting natural images [37], [21].”
DAG-Recurrent Neural Networks for Scene Labeling (2015, methodology): “As an example, the frequency ratio between the most frequent (sky) and the most rare class (moon) on the SiftFlow dataset is 3 .”
Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs (2013, methodology): “The SIFT-flow dataset [24] is dominated by “stuff” with a small proportion of “things” pixels.”
Region-based Segmentation and Object Detection (2009, methodology): “[14] use a non-parametric approach to image labeling by warping a given image onto a large set of labeled images and then combining the results.”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation (2011, positive): “For the latter comparison, we used the code provided on the authors’ website, which corresponds to their far more efficient coarse-to-fine version from [22].”
STC: A Simple to Complex Framework for Weakly-Supervised Semantic Segmentation (2015, methodology): “These methods are usually applied on simple or small scale datasets, e.g. , MSRA [38] and SIFT-ﬂow [39].”
Integrating parametric and non-parametric models for scene labeling (2015, methodology): “Four global features are compared in our experiment: GIST [19] is a global summary of scene images that captures scene structure and layout; SIFT-SPM (GT) [14] is pooled from low-level local SIFT [17] (ground truth label map [16]) in a 3(2)-layer spatial pyramid.”
Figure-ground segmentation by transferring window masks (2012, background): “As these windows exhibit less variability than whole images and are often centered on objects, they form much better support regions for segmentation transfer.”
Superparsing (2010, methodology): “However, unlike the approach of [15], which works best if the retrieval set images are very similar to the test image in terms of spatial layout of the classes, we transfer labels at the level of superpixels , or coherent image regions produced by a bottom-up segmentation method.”
Weakly-Supervised Dual Clustering for Image Semantic Segmentation (2013, methodology): “Fully-supervised methods [21, 24, 11] and weaklysupervised methods [27, 26] are used as compared methods and the condition settings of them are displayed in Table 4.”
Towards Unified Human Parsing and Pose Estimation (2014, methodology): “Similar to the influential deformable part models [6], some methods [14] treat the entire body as a mixture of templates.”
A Data-Driven Approach for Event Prediction (2010, neutral): “This idea has proven robust in many non-parametric data-driven techniques such as label transfer [17] and scene completion [18] amongst many others.”
Action recognition by dense trajectories (2011, background): “Experimental results show a significant improvement over the state of the art on four datasets of varying difficulty, i.e. KTH, YouTube, Hollywood2 and UCF sports.”