SUN Database: Exploring a Large Collection of Scene Categories

By Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, Aude Oliva. 2014

509 citations recorded by the source corpus.

Progress in scene understanding requires reasoning about the rich and diverse visual environments that make up our daily experience. To this end, we propose the Scene Understanding database, a nearly exhaustive collection of scenes categorized at the same level of specificity as human discourse. The database contains 908 distinct scene categories and 131,072 images. Given this data with both scene and object labels available, we perform in-depth analysis of co-occurrence statistics and the contextual relationship. To better understand this large scale taxonomy of scene categories, we perform two human experiments: we quantify human scene recognition accuracy, and we measure how typical each image is of its assigned scene category. Next, we perform computational experiments: scene recognition with global image features, indoor versus outdoor classification, and “scene detection,” in which we relax the assumption that one image depicts only one scene category. Finally, we relate human experiments to machine performance and explore the relationship between human and machine recognition errors and the relationship between image “typicality” and machine recognition accuracy.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Evaluation & Benchmarks

Citations in the corpus

Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, methodology): “We follow the experimental setup in [1, 15], which uses a nonstandard train/test split since it is computationally expensive to run all of our experiments on the 10 standard subsets proposed by [52].”
PlaNet - Photo Geolocation with Convolutional Neural Networks (2016, methodology): “The SUN survey paper [50] shows that Over-feat [44], a CNN trained on ImageNet [9] images, consistently outperforms other approaches, including global descriptors like GIST and local descriptors like SIFT, motivating our use of CNNs for image geolocation.”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “These need to be merged a posteriori [50, 58].”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “, 2014), SUN397 (Xiao et al., 2016), Stanford Cars (Krause et al.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “[85] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.”
Object Detectors Emerge in Deep Scene CNNs (2014, background): “In this section, our goal is to understand the nature of the representation that the network is learning.”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, methodology): “[38, 28] It is also different than most object recognition tasks, where the goal is to label everything visible in an image.”
Material recognition in the wild with the Materials in Context Database (2014, background): “Recently, large-scale datasets such as ImageNet [21], SUN [32], and SUN Attributes [19] combined with convolutional neural networks (CNNs) have driven rapid improvements in object recognition and scene classiﬁcation.”
From generic to specific deep representations for visual recognition (2014, neutral): “non-ConvNet [35] [22] [40] [26] [38] [45] [25] [12] [18] [24] [42] [30] [37] [46] [37] [37] [4] 71.”
Playing for Benchmarks (2017, background): “The SUN and Places datasets support research on scene recognition [62, 66].”
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision (2017, positive): “…model, we use them as inputs to linear classifiers on the following test datasets: the MIT indoor scenes dataset [11], the SUN scene dataset [18], the CUB birds dataset [17], the Oxford flowers dataset [10], and Stanford 40 action recognition Model Train Loss Test Loss q@1 q@5 q@10 p@1 p@5…”
Robust fine-tuning of zero-shot models (2021, methodology): “On the y -axis we display in-distribution accuracy improvements of WiSE-FT averaged over seven datasets (Deng et al., 2009; Krizhevsky et al., 2009; Cimpoi et al., 2014; Bossard et al., 2014; Xiao et al., 2016; Krause et al., 2013).”
YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video (2017, background): “They vary in detail from bounding boxes (Caltech-101 [14], MIT-CSAIL [51], ImageNet [9, 41], PASCAL VOC [12, 13], SUN2012 [55, 56]) to pixel-level segmentations (Berkeley Segmentation Data Set [32], Caltech-101 [14], PASCAL VOC [12, 13], Microsoft COCO [31]).”
A Survey of the Four Pillars for Small Object Detection: Multiscale Representation, Contextual Information, Super-Resolution, and Region Proposal (2022, methodology): “Thus, this dataset extracted purely ten categories of small objects from MSCOCO and Scene Understanding database [61], containing approximately 8393 object instances and 4925 images.”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, background): “Their details (Deng et al., 2009; Zhou et al., 2017; Xiao et al., 2016; Krizhevsky & Hinton, 2009; Krause et al., 2013; Nilsback & Zisserman, 2010) are described in Table 1.”
Factors of Transferability for a Generic ConvNet Representation (2014, methodology): “ (source task): Results with the ConvNet representation optimized for different amount of training data. First row shows the results when the network is trained on scene recognition dataset of SUN397 [44] dataset with 100K images. The second row corresponds to the network trained on Places dataset [”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, background): “In addition, we fine-tune and evaluate on eight diverse datasets where zero-shot models perform poorly [31, 55]: Cars [41], DTD [14], EuroSAT [21], GTSRB [69], MNIST [44], RESISC45 [13], SUN397 [79], and SVHN [50].”
The New Data and New Challenges in Multimedia Research (2015, result): “In contrast to the ImageNet dataset, COCO has fewer categories but more instances per category.”