Representation Learning via Invariant Causal Mechanisms

By Jovana Mitrovi'c, Brian McWilliams, Jacob Walker, Lars Buesing, Charles Blundell. 2020

290 citations recorded by the source corpus.

Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success of these methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on $51$ out of $57$ games.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “Data augmentation techniques are widely used in computer vision tasks [21, 40] and have recently been applied to deep RL tasks [18, 20, 33, 26].”
Self-Supervised Learning: Generative or Contrastive (2020, background): “ReLIC [82] argues that contrastive pre-training teaches the encoder to causally disen-”
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss (2021, background): “Several other works (Bansal et al., 2020, Mitrovic et al., 2020, Tian et al., 2020b, Tsai et al., 2020, Wang and Isola, 2020) also theoretically study self-supervised learning.”
Encoders and Ensembles for Task-Free Continual Learning (2021, methodology): “However, our best results were obtained using the ReLIC contrastive method, where we closely followed the setup in [33], using the same hyperparameters and optimisation settings.”
Contrastive Learning With Stronger Augmentations (2021, methodology): “Concurrently, KL loss for consistency learning is widely used to help representation learning, such as CO2 [29] and RELIC [30], where regularization is added to enforce the consistency between the embeddings from different data augmentation.”
Bootstrapped Representation Learning on Graphs (2021, background): “They have displayed performance that matches or improves over the equivalent methods trained with labeled data (Tian et al., 2020; Bachman et al., 2019; Mitrovic et al., 2021; Caron et al., 2020; Xu et al., 2020).”
CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2022, methodology): “Since the targets X? are unknown, we construct a proxy contrastive learning task inspired by Mitrovic et al. (2020).”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “The advancement of SSL [3], [4], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] has exhibitedrapidprogress, capturing signiﬁcant attention within the research community (Fig.”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “single-source domain generalization, [107, 135] used data augmentation to present information of the causal factor.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “However, ReLIC assumes independence between content and style and does not address identifiability.”