MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation

By John Lambert, Zhuang Liu, Ozan Sener, James Hays, Vladlen Koltun. 2020

241 citations recorded by the source corpus.

We present MSeg, a composite dataset that unifies se- mantic segmentation datasets from different domains. A naive merge of the constituent datasets yields poor performance due to inconsistent taxonomies and annotation practices. We reconcile the taxonomies and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images. The resulting composite dataset enables training a single semantic segmentation model that functions effectively across domains and generalizes to datasets that were not seen during training. We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model’s robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions. A model trained on MSeg ranks first on the WildDash leaderboard for robust semantic segmentation, with no exposure to WildDash data during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “In MSeg [40], the authors manually merge different datasets and train a more generalized model on the composite set, which is still limited to being a closed set.”
Language-driven Semantic Segmentation (2022, methodology): “We finally train LSeg on a mix of 7 different datasets (Lambert et al., 2020), including ADE20K (Zhou et al., 2019), BDD (Yu et al., 2020), Cityscapes (Cordts et al., 2016), COCO-Panoptic (Lin et al., 2014; Caesar et al., 2018), IDD (Varma et al., 2019), Mapillary Vistas (Neuhold et al., 2017), and…”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, unknown): “This underscores research signiﬁcance in ﬁnding an efﬁcient way to distill prior from other types of annotation.”
Enhancing Photorealism Enhancement (2021, methodology): “It consists of pretrained robust segmentation (MSeg [74]) and perceptual (VGG [95]) networks (green).”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Compared with K-Net and MaskFormer, both K-Net+ and MaskFormer+ achieve over 3-4% improvements due to the usage of stronger neck [25], which close the gaps between their original results and Mask2Former.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “We perform additional cross-dataset evaluation using datasets in MSeg [28] in Appendix D. Figure 4 shows 6 manually selected proposals to demonstrate our model can organize images into semantically meaningful regions.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “We first compare our approach to the most closely related work on zero-shot 3D semantic segmentation: MSeg [29] Voting and 3DGenz [39].”
Simple Multi-dataset Detection (2021, methodology): “MSeg [19] manually uniﬁed the taxonomies of 7 semantic segmentation datasets and used Amazon Mechanical Turk to resolve inconsistent annotations between datasets.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Sometimes, MTL papers take mix datasets for a “single” task and consider each dataset as a different task [37, 47, 35, 46].”
LMSeg: Language-guided Multi-dataset Segmentation (2023, methodology): “For multi-dataset semantic segmentation, MSeg (Lambert et al., 2020) manually creates a common taxonomy to unite segmentation datasets from multiple domains.”
Open-Vocabulary Image Segmentation (2021, methodology): “In this experiments, we use the setup and the curated annotations in MSeg [27].”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, background): “Recently, various natural language problems and computer vision problems [65, 38, 29] have achieved impressive progress with large-scale data training.”