Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization Without Accessing Target Domain Data

By Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto L. Sangiovanni-Vincentelli, Kurt Keutzer, Boqing Gong. IEEE International Conference on Computer Vision, 2019

472 citations recorded by the source corpus.

We propose to harness the potential of simulation for semantic segmentation of real-world self-driving scenes in a domain generalization fashion. The segmentation network is trained without any information about target domains and tested on the unseen target domains. To this end, we propose a new approach of domain randomization and pyramid consistency to learn a model with high generalizability. First, we propose to randomize the synthetic images with styles of real images in terms of visual appearances using auxiliary datasets, in order to effectively learn domain-invariant representations. Second, we further enforce pyramid consistency across different "stylized" images and within an image, in order to learn domain-invariant and scale-invariant features, respectively. Extensive experiments are conducted on generalization from GTA and SYNTHIA to Cityscapes, BDDS, and Mapillary; and our method achieves superior results over the state-of-the-art techniques. Remarkably, our generalization results are on par with or even better than those obtained by state-of-the-art simulation-to-real domain adaptation methods, which access the target domain data at training time.

View Paper: https://www.semanticscholar.org/paper/1aab4548a7d55cf8a3562f8710b50ed20ad32546

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space (2021, background): “Domain generalization [5, 9, 12, 14, 43, 47, 58, 59] aims to learn a model from multiple source domains such that it can directly generalize to unseen target domains.”
Domain Generalization: A Survey (2021, methodology): “patient’s data in advance [19]; in traffic scene semantic segmentation it is infeasible to collect data capturing all different scenes and under all possible weather conditions [20];”
Synthetic Data for Deep Learning (2019, background): “In particular, recent works [357, 680] have introduced the so-called pyramid consistency loss instead of C (p(xT ), p̂(xT )) that tries to enforce consistency across domains on the activation maps of later layers of the network.”
Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization (2020, methodology): “on methods proposed to generate synthetic images derived from the given multiple source domains to increase the diversity of the input images, so that the network could learn from a larger data space [49,50]. Another promising direction is to extract domain-invariant features over multiple source do”
Learning to Generate Novel Domains for Domain Generalization (2020, unknown): “ure of objects, background scenes, lighting conditions, etc. via a computer simulator [49]. Recently, DR has been successfully used in some computer vision applications, such as semantic segmentation [54,55] and vehicle detection for autonomous driving [42]. However, our approach is signicantly dier”
Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation (2020, methodology): “Since DRPC used 16 domains, it requires at least 16 times more memory and computing power.”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “…on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43, 60, 53, 6, 69, 18, 58, 37, 65] have been proposed in semantic segmentation, but only a few DG studies [64, 44] address semantic segmentation, as the majority of the DG methods mainly focused on image classiﬁcation.”
Deep Domain-Adversarial Image Generation for Domain Generalisation (2020, background): “We assume that augmenting the original training data of source domains with synthetic data from unseen domains could make the task model intrinsically more domain-generalisable (Tobin et al. 2017; Yue et al. 2019).”
FSDR: Frequency Space Domain Randomization for Domain Generalization (2021, methodology): “Without identifying and preserving DIFs, state-of-theart methods [77, 14] tend to over-randomize images which may degrade image structures and semantics and lead to sup-optimal network generalization.”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “Similar techniques were also used in [29], [33], [34], [35] to strengthen the generalization capability”