Spatial pyramid co-occurrence for image classification

By Yi Yang, Shawn D. Newsam. 2011

323 citations recorded by the source corpus.

We describe a novel image representation termed spatial pyramid co-occurrence which characterizes both the photometric and geometric aspects of an image. Specifically, the co-occurrences of visual words are computed with respect to spatial predicates over a hierarchical spatial partitioning of an image. The representation captures both the absolute and relative spatial arrangement of the words and, through the choice and combination of the predicates, can characterize a variety of spatial relationships. Our representation is motivated by the analysis of overhead imagery such as from satellites or aircraft. This imagery generally does not have an absolute reference frame and thus the relative spatial arrangement of the image elements often becomes the key discriminating feature. We validate this hypothesis using a challenging ground truth image dataset of 21 land-use classes manually extracted from high-resolution aerial imagery. Our approach is shown to result in higher classification rates than a non-spatial bagof- visual-words approach as well as a popular approach for characterizing the absolute spatial arrangement of visual words, the spatial pyramid representation of Lazebnik et al. [7]. While our primary objective is analyzing overhead imagery, we demonstrate that our approach achieves state-of-the-art performance on the Graz-01 object class dataset and performs competitively on the 15 Scene dataset.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Junwei Han
2. Gong Cheng
3. Liangpei Zhang
4. Gui-Song Xia
5. Xiaoqiang Lu
6. Yanfei Zhong
7. Bo Du
8. Xiang Bai

In Media Technology

1. Liangpei Zhang
2. Gong Cheng
3. Junwei Han
4. Xiaoqiang Lu
5. Yanfei Zhong
6. Bo Du
7. Gui-Song Xia
8. Lei Guo

Citations in the corpus

• Remote Sensing Image Scene Classification: Benchmark and State of the Art (2017, methodology): “they are generally used as building blocks to construct global image features, such as the well-known bag-of-visual-words (BoVW) models [6, 8, 9, 14, 19, 29, 36, 38, 39, 55, 93, 101, 122, 123] and HOG feature-based part models [22, 23, 27, 103].”
• Learning coarse-to-fine sparselets for efficient object detection and scene classification (2015, methodology): “Different from PASCAL VOC 2007 dataset [22] and MIT Scene-67 dataset [23] in which images are all natural scene (non-overhead view) images, this is a new yet challenging high-spatial-resolution overhead images dataset collected by [26], which consists of 21 aerial scenes with complex geospatial obje”
• Using convolutional features and a sparse autoencoder for land-use scene classification (2016, methodology): “In the BOVW context, Yang and Newsam (2011) proposed another method, which is noted as spatial pyramid co-occurrence, where a BOVW histogram involved both absolute and relative spatial arrangements of the visual words whose co-occurrence was counted over a spatial partitioning of the target image.”
• Remote Sensing Image Classification: No Features, No Clustering (2015, methodology): “The two methods we choose for comparison are spatial pyramid co-occurrence [19], [47] and the unsupervised feature learning method [48].”
• Land-Use Scene Classification in High-Resolution Remote Sensing Images Using Improved Correlatons (2015, methodology): “For the land-use classification in HRIs, Yang and Newsam [15] proposed a spatial co-occurrence kernel to explore spatial extensions to the BoVW approach.”
• Land Use Classification in Remote Sensing Images by Convolutional Neural Networks (2015, methodology): “Improved versions can be obtained by simply combining SCK and SPCK with the BOVW model (SCK+BOVW and SPCK+, respectively) or SPCK with SPMK (SPCK++) [14], [17].”
• PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval (2017, methodology): “In [86], the spatial pyramid co-occurrence kernel (SPCK) is proposed for image classification by integrating the absolute and relative spatial information that is ignored in the standard BOVW framework.”
• Remote Sensing Image Scene Classification Using Multi-Scale Completed Local Binary Patterns and Fisher Vectors (2016, background): “Thus, a spatial co-occurrence kernel, which is general enough to characterize a variety of spatial arrangements, was proposed in [3] to capture both the absolute and relative spatial layout of an image.”
• Gabor-Filtering-Based Completed Local Binary Patterns for Land-Use Scene Classification (2015, background): “Although the BoW model has demonstrated the effectiveness for the remotely sensed land-use scene classification [1, 3], it ignores spatial relationships of the local features.”
• Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks (2016, background): “Digital Object Identifier 10.1109/LGRS.2015.2499239 (SPMK) [1], spatial pyramid cooccurrence kernel (SPCK ++ ) [2], min-tree kd-tree [3], and sparse coding [4] methods.”
• A Color-Texture-Structure Descriptor for High-Resolution Satellite Image Classification (2016, methodology): “Methods Classification Results BOVW [20] 71.”
• Strathprints Institutional Repository Shuhui and Ren, Jinchang (2015) Effective and Efficient Midlevel Visual Elements-oriented Land-use Classification Using Vhr Remote Sensing Images. Ieee Transactions on Geoscience and Remote Sensing, 53 (8). Pp. 4238-4249. Issn 0196-2892 Effective and Efficient M (unknown): “1 as examples, which are from a publicly available 21 LULC classes data set 1 [6] [7].”
• Semantic Annotation of High-Resolution Satellite Images via Weakly Supervised Learning (2016, methodology): “In recent years, the bag of visual words (BOVW) model [28] has successfully served as an intermediate feature representation for land-use classification [9], [10], [29]–[32].”
• The Fisher Kernel Coding Framework for High Spatial Resolution Scene Classification (2016, methodology): “During the training of the SVM classifier, the histogram intersection kernel (HIK) is adopted due to its performance in image classification [44].”
• Land-use scene classification using multi-scale completed local binary patterns (2016, methodology): “The BOVW model has been successfully applied to remote sensing land-use scene classification and achieved good performance [2,3].”