COCO-Stuff: Thing and Stuff Classes in Context

By Holger Caesar, Jasper R. R. Uijlings, Vittorio Ferrari. 2016

1,715 citations recorded by the source corpus.

Semantic classes can be either things (objects with a well-defined shape, e.g. car, person) or stuff (amorphous background regions, e.g. grass, sky). While lots of classification and detection works focus on thing classes, less attention has been given to stuff classes. Nonetheless, stuff classes are important as they allow to explain important aspects of an image, including (1) scene type; (2) which thing classes are likely to be present and their location (through contextual reasoning); (3) physical attributes, material types and geometric properties of the scene. To understand stuff and things in context we introduce COCO-Stuff1, which augments all 164K images of the COCO 2017 dataset with pixel-wise annotations for 91 stuff classes. We introduce an efficient stuff annotation protocol based on superpixels, which leverages the original thing annotations. We quantify the speed versus quality trade-off of our protocol and explore the relation between annotation time and boundary complexity. Furthermore, we use COCO-Stuff to analyze: (a) the importance of stuff and thing classes in terms of their surface cover and how frequently they are mentioned in image captions; (b) the spatial relations between stuff and things, highlighting the rich contextual relations that make our dataset unique; (c) the performance of a modern semantic segmentation method on stuff and thing classes, and whether stuff is easier to segment than things.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Top: Samples from our LDM for layout-to-image synthesis on COCO [4].”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “For semantic segmentation, we use visual-transformers to re-design the feature-pyramid networks (FPN) and save 6.4x MACs while achieving higher mIoU on LIP and COCO-Stuff datasets.”
Semantic Understanding of Scenes Through the ADE20K Dataset (2016, methodology): “YU Depth V2 [23], SUN database [34], SUN RGB-D dataset [31], CityScapes Semantic Understanding of Scenes through the ADE20K Dataset 3 dataset [7], and OpenSurfaces [2, 3]. Recently COCO stuff dataset [4] provides additional stuff segmentation complementary to the 80 object categories in COCO dataset”
High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “First, we evaluate our model’s ability to compete with state-of-the-art methods in layout-to-image on COCO [5] and Visual Genome [1] on resolutions up to 256 pixels.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “In each round, the user is given a pair of generated images of SPADE (Park et al., 2019) and ours using the same validation input.”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “4.1 (i.e. image size 256 × 256, latent size 16 × 16), we perform various conditional image synthesis experiments: (i): Semantic image synthesis, where we condition on semantic segmentation masks of ADE20K [72], a webscraped landscapes dataset (S-FLCKR) and COCO-Stuff [6].”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “3 EXPERIMENTS We test our LostGAN in the COCO-Stuff dataset [22] and the Visual Genome (VG) dataset [50].”
Object-Centric Image Generation from Layouts (2020, methodology): “Building upon MS-COCO, the COCO-Stuff [5] dataset augments the MS-COCO dataset with pixel level stuff annotations.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “• COCO-Stujf [6] is derived from the COCO dataset [31].”
StyleGAN-NADA (2021, methodology): “In this setup, we utilize a model pre-trained on the COCO-stuff dataset [9], and aim to change one of the model classes to a novel class which shares the shape of the source class, but differs in texture.”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “COCO-Stuff [6] is a challenging and diverse segmentation dataset containing “stuff” classes ranging from buildings to bodies of water.”
Language-driven Semantic Segmentation (2022, background): “We finally train LSeg on a mix of 7 different datasets (Lambert et al., 2020), including ADE20K (Zhou et al., 2019), BDD (Yu et al., 2020), Cityscapes (Cordts et al., 2016), COCO-Panoptic (Lin et al., 2014; Caesar et al., 2018), IDD (Varma et al., 2019), Mapillary Vistas (Neuhold et al., 2017), and ”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “We evaluate MaskFormer on five semantic segmentation datasets with various numbers of categories: Cityscapes [15] (19 classes), Mapillary Vistas [34] (65 classes), ADE20K [55] (150 classes), COCOStuff-10K [3] (171 classes), and ADE20K-Full [55] (847 classes).”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Detection and Segmentation Datasets Many datasets for image segmentation have been proposed with the goal of densely recognizing general objects and “stuff” in image scenes like street view [7, 17, 61], natural scenes [9, 20, 35, 44,65], and indoor spaces [13,19,53] 1 for comparison).”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “Models based on Fully Convolutional Networks (FCNs) [64, 49] have demonstrated significant improvement on several segmentation benchmarks [17, 52, 13, 83, 5].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “For the task of semantic segmentation [20, 63, 14, 97, 7], we consider two challenges in applying Deep Convolutional Neural Networks (DCNNs) [50].”
Hybrid Task Cascade for Instance Segmentation (2019, methodology): “f 2017train (115k images) and report results on 2017val and 2017test-dev. Typical instance annotations are used to supervise box and mask branches, and the semantic branch is supervised by COCO-stuff [4] annotations. Evaluation Metrics. We report the standard COCO-style Average Precision (AP) metric”
Incremental Learning in Semantic Segmentation from Image Labels (2021, positive): “We follow the training split and the annotation of [7] that solves the overlapping annotation problem present in [40].”