Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss

By Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, Tengyu Ma. Neural Information Processing Systems, 2021

402 citations recorded by the source corpus.

Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm, which learns representations by pushing positive pairs, or similar examples from the same class, closer together while keeping negative pairs far apart. Despite the empirical successes, theoretical foundations are limited -- prior analyses assume conditional independence of the positive pairs given the same class label, but recent empirical applications use heavily correlated positive pairs (i.e., data augmentations of the same image). Our work analyzes contrastive learning without assuming conditional independence of positive pairs using a novel concept of the augmentation graph on data. Edges in this graph connect augmentations of the same data, and ground-truth classes naturally form connected sub-graphs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective on neural net representations. Minimizing this objective leads to features with provable accuracy guarantees under linear probe evaluation. By standard generalization bounds, these accuracy guarantees also hold when minimizing the training contrastive loss. Empirically, the features learned by our objective can match or outperform several strong baselines on benchmark vision datasets. In all, this work provides the first provable analysis for contrastive learning where guarantees for linear probe evaluation can apply to realistic empirical settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3f1774d91c7c219f063f40b894e837e6c48b2bb2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CaSpeR: Latent Spectral Regularization for Continual Learning (2023, background): “The thorough study in [21] interprets contrastive learning as a parametric form of spectral clustering on the input augmentation graph, which points to a link with our approach.”
A Cookbook of Self-Supervised Learning (2023, background): “In fact, attempts at unifying SSL methods under a single viewpoint have only started to emerge in the last year [HaoChen et al., 2021, Balestriero and LeCun, 2022, Shwartz-Ziv et al., 2022, Garrido et al., 2022b].”