Domain Generalization by Solving Jigsaw Puzzles

By Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, Tatiana Tommasi. 2019

952 citations recorded by the source corpus.

Human adaptability relies crucially on the ability to learn and merge knowledge both from supervised and unsupervised learning: the parents point out few important concepts, but then the children fill in the gaps on their own. This is particularly effective, because supervised learning can never be exhaustive and thus learning autonomously allows to discover invariances and regularities that help to generalize. In this paper we propose to apply a similar approach to the task of object recognition across domains: our model learns the semantic labels in a supervised fashion, and broadens its understanding of the data by learning from self-supervised signals how to solve a jigsaw puzzle on the same images. This secondary task helps the network to learn the concepts of spatial correlation while acting as a regularizer for the classification task. Multiple experiments on the PACS, VLCS, Office-Home and digits datasets confirm our intuition and show that this simple method outperforms previous domain generalization and adaptation solutions. An ablation study further illustrates the inner workings of our approach.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “We use ? to denote the methods that use the training setting in (Carlucci et al., 2019) (e.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Prior pretext tasks include image colorization [9, 30, 38, 39, 86, 87], orientation prediction [20], affine transform prediction [85], predicting contextual image patches [10], reordering image patches [5, 21, 53, 54, 56], counting visual primitives [55], or their combinations [11].”
In Search of Lost Domain Generalization (2020, methodology): “Carlucci et al. [2019a] constructs an auxiliary classiﬁcation task aimed at solving jigsaw puzzles of image patches.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Other techniques for domain generalization include conditional variance regularization (Heinze-Deml & Meinshausen, 2017); self-supervision (Carlucci et al., 2019); and meta-learning-based approaches (Li et al.”
Gradient Matching for Domain Generalization (2021, positive): “, 2020), matching mean and variance of feature distributions across domains (Sun and Saenko, 2016), learning useful representations by solving Jigsaw puzzles (Carlucci et al., 2019) or using the maximum mean discrepancy (Gretton et al.”
Reducing Domain Gap by Reducing Style Bias (2021, methodology): “We evaluate the efﬁcacy of SagNets against recent DG methods including D-SAM [9], JiGen [4], Epi-FCR [30], MASF [8], We split the training data of PACS into 70% training and 30% validation following the ofﬁcial split [29], and Ofﬁce-Home into 90% training and 10% validation following [9].”
Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer (2020, background): “In addition, pretext tasks can also be jointly trained with supervised learning tasks on labeled data with shared weights like in [67], [68].”
Test-Time Training for Out-of-Distribution Generalization (2019, methodology): “The same idea is used in few-shot learning (Su et al., 2019), and domain generalization (Carlucci et al., 2019).”
Empirical Bayes Transductive Meta-Learning with Synthetic Gradients (2020, methodology): “We follow the standard experimental setting (Carlucci et al., 2019), where the feature network is implemented by ResNet-18.”
Domain Generalization with MixStyle (2021, methodology): “These include domain alignment-based CCSA (Motiian et al., 2017) and MMD-AAE (Li et al., 2018b), Jigsaw puzzle-based JiGen (Carlucci et al., 2019), adversarial gradient-based CrossGrad (Shankar et al., 2018), meta-learning-based Metareg (Balaji et al., 2018) and Epi-FCR (Li et al., 2019), and data…”
Domain Generalization Using a Mixture of Multiple Latent Domains (2019, methodology): “Following the previous work (Carlucci et al. 2019), we use three domains as the source domain, and the other as the target.”
FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space (2021, methodology): “Unseen Site A B C D E F Average A B C D E F Average Dice Coefficient (Dice) ↑ Hausdorff Distance (HD) ↓ JiGen [3] 89.95 85.81 84.06 87.34 81.32 89.11 86.26 10.51 11.53 11.70 11.49 14.80 9.02 11.51 BigAug [60] 89.63 84.62 83.86 87.66 81.20 88.96 85.99 10.68 11.78 12.07 10.66 13.98 9.73 11.48 Epi-FCR ”
When Does Self-supervision Improve Few-shot Learning? (2019, neutral): “[6] used self-supervision to improve domain generalization.”
Reducing Domain Gap via Style-Agnostic Networks (2019, methodology): “We evaluate the efficacy of SagNets against state-of-the-art DG methods including D-SAM [9], JiGen [5], Epi-FCR [31], MASF [8], and MMLD [37].”
Adversarial target-invariant representation learning for domain generalization (2019, methodology): “We compare the proposed approach against current state-of-the-art methods, namely, the recently proposed Epi-FCR (Carlucci et al., 2019) and JiGen (Li et al., 2019).”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, methodology): “We use the exact same architecture, training procedure and data augmentation strategy as the state-of-the-art Jigsaw Puzzle approach (Carlucci et al., 2019) (except with IRM or V-REx intead of JigSaw as auxilliary loss) for all three methods, experimental details as well as hyperparameter selection…”