Semantic Understanding of Scenes Through the ADE20K Dataset

By Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, Antonio Torralba. 2016

2,400 citations recorded by the source corpus.

Semantic understanding of visual scenes is one of the holy grails of computer vision. Despite efforts of the community in data collection, there are still few image datasets covering a wide range of scenes and object categories with pixel-wise annotations for scene understanding. In this work, we present a densely annotated dataset ADE20K, which spans diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. Totally there are 25k images of the complex everyday scenes containing a variety of objects in their natural spatial context. On average there are 19.5 instances and 10.5 object classes per image. Based on ADE20K, we construct benchmarks for scene parsing and instance segmentation. We provide baseline performances on both of the benchmarks and re-implement state-of-the-art models for open source. We further evaluate the effect of synchronized batch normalization and find that a reasonably large batch size is crucial for the semantic segmentation performance. We show that the networks trained on ADE20K are able to segment a wide variety of scenes and objects.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “We also run experiments on the widely-used ADE20K [77] dataset.”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “We evaluate BEIT on the ADE20K benchmark (Zhou et al., 2019) with 25K images and 150 semantic categories.”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “4.1 (i.e. image size 256 × 256, latent size 16 × 16), we perform various conditional image synthesis experiments: (i): Semantic image synthesis, where we condition on semantic segmentation masks of ADE20K [72], a webscraped landscapes dataset (S-FLCKR) and COCO-Stuff [6].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “We use the training hyperparameters from [78], i.”
Segment Anything (2023, background): “We observe that SA-1B has greater coverage of image corners compared to LVIS v1 [44] and ADE20K [117], the two most similarly distributed datasets, while COCO [66] and Open Images V5 [60] have a more prominent center bias.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “As shown in Table 3, we use VLM O as an image-only encoder and evaluate it on image classiﬁcation (ImageNet [38]) and semantic segmentation (ADE20K [50]) tasks.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Swin-T with the shifted window partitioning outperforms the counterpart built on a single window partitioning at each stage by +1.1% top-1 accuracy on ImageNet-1K, +2.8 box AP/+2.2 mask AP on COCO, and +2.8 mIoU on ADE20K.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “…of models pre-trained using ImageNet + dataset, we evaluate on following tasks: (1) semantic segmentation with DeepLabv3 [11] on the ADE20K dataset [74], (2) object detection with Mask-RCNN [20] on the MS-COCO dataset [34], and (3) fine-grained classification on the CIFAR-100 [31], Flowers-102…”
Visual Prompt Tuning (2022, methodology): “We explore the feasibility of VPT beyond visual classification, by evaluating ADE20K [90] semantic segmentation task with a Transformer model, SETR-PUP [89].”
Language-driven Semantic Segmentation (2022, methodology): “We ﬁnally train LSeg on a mix of 7 different datasets (Lambert et al., 2020), including ADE20K (Zhou et al., 2019), BDD (Yu et al., 2020), Cityscapes (Cordts et al., 2016), COCO-Panoptic (Lin et al., 2014; Caesar et al., 2018), IDD (Varma et al., 2019), Mapillary Vistas (Neuhold et al., 2017), and S”
K-Net: Towards Unified Image Segmentation (2021, methodology): “To show the effectiveness of the proposed unified framework on different segmentation tasks, we conduct extensive experiments on COCO dataset [38] for panoptic and instance segmentation, and ADE20K dataset [72] for semantic segmentation.”
Pyramid Scene Parsing Network (2016, methodology): “We hope the implementation details publicly available can help the community adopt these useful strategies for scene parsing and semantic segmentation and advance related techniques.”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, methodology): “State-of-the-art segmentations are typically obtained by combining classification models pretrained on ImageNet (typically referred to as body) with segmentation-specific head architectures and jointly fine-tuning them on suitable, (densely) annotated training data like Cityscapes [5], COCO-Stuff [1”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “When training a single model on the train+validation set with the default 160,000 iterations, our method ranks 1 st place in the highly competitive ADE20K test server leaderboard.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “To show the effectiveness of our approach, we carry out comprehensive experiments on seven public datasets, which include six popular datasets for semantic segmentation on indoors and outdoors scenes (NYUDv2, PASCAL VOC 2012, SUN-RGBD, PASCAL-Context, Cityscapes, ADE20K MIT), and one dataset for obj”