Contrastive Learning Inverts the Data Generating Process

By Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, Wieland Brendel. 2021

283 citations recorded by the source corpus.

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “, 2020), in which identifiability has been used as an analysis tool (Roeder et al., 2021; Zimmermann et al., 2021; Von Kügelgen et al., 2021).”
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss (2021, background): “Zimmermann et al. (2021) prove that for a speciﬁc data generating process, contrastvie learning can learn representations that recover the latent variable.”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “Causal representation learning Finally, the ideas here connect to causal representation learning (e.g., Higgins et al., 2016; Hyvarinen & Morioka, 2016; Higgins et al., 2018; Khemakhem et al., 2020; Zimmermann et al., 2021; Sch¨olkopf et al., 2021; Moran et al., 2021; Wang et al., 2023).”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, methodology): “(3) Our setting for modelling data augmentation differs from that commonly assumed in (multi-view) disentanglement and ICA in that we do not assume that the latent factors z = (c, s) are mutually (or conditionally) independent, i.e., we allow for arbitrary (non-factorised) marginals pz in (2).6 Caus”
The Platonic Representation Hypothesis (2024, unknown): “Nonetheless, similar arguments have been made theoretically and empirically that cooccurrence relations are learned by practical contrastive (Wang & Isola, 2020; Zimmermann et al., 2021) and predictive learners (Papyan et al., 2020; Roeder et al., 2021).”
Covariance Structure and Coordinate Heterogeneity Govern Binary Quantization of Contrastive Embeddings (2026, unknown): “This framework underlies SimCLR (Chen et al., 2020), MoCo (He et al., 2020), CLIP (Radford et al., 2021), and has been studied through downstream guarantees (Saunshi et al., 2019), spectral analysis (HaoChen et al., 2021), and identifiability (Zimmermann et al., 2021).”
High-Dimension Human Value Representation in Large Language Models (2024, background): “In the value embedding learning, the core challenge lies in the fact that ϑ value exists as a latent variable [74, 75].”