On Mutual Information Maximization for Representation Learning

By Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, Mario Lucic. 2019

567 citations recorded by the source corpus.

Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective for representation learning may lead to highly entangled representations due to its invariance under arbitrary invertible transformations. Nevertheless, these methods have been repeatedly shown to excel in practice. In this paper we argue, and provide empirical evidence, that the success of these methods cannot be attributed to the properties of MI alone, and that they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parametrization of the employed MI estimators. Finally, we establish a connection to deep metric learning and argue that this interpretation may be a plausible explanation for the success of the recently introduced methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “However, it is not clear if the success of contrastive approaches is determined by the mutual information, or by the speciﬁc form of the contrastive loss (Tschannen et al., 2019).”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Both were shown to be weakly correlated with representation utility (Locatello et al., 2019; Tschannen et al., 2019).”
What makes for good views for contrastive learning (2020, background): “Such connection has been discussed further in [45, 56].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “A recent study also shows that better performance of instance discrimination could worsen the performance on downstream tasks [12].”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “We attribute this to the fact that the InfoMax critics are simple MLP networks with only 1 hidden layer, which is sufficient for achieving good representations in practice [17].”
Supervised Contrastive Learning (2020, methodology): “The state of the art family of models for self-supervised representation learning using this paradigm are collected under the umbrella of contrastive learning [54, 18, 22, 48, 43, 3, 50].”
Self-Supervised Learning: Generative or Contrastive (2020, background): “cess, some recent studies [15], [18], [47], [120] cast doubt on”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, background): “A recent study has questioned whether the success of InfoNCE as an objective function is due to its property as a lower bound on mutual information and provides an alternative hypothesis based on metric learning (Tschannen et al., 2019).”
Better Fine-Tuning by Reducing Representational Collapse (2020, background): “One approach would be to estimate mutual information of representations across tasks before and after ﬁne-tuning, but estimation of mutual information is notoriously hard, especially in high-dimensions (Tschannen et al., 2019).”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, neutral): “While contrastive learning in aid of model-free RL has been studied in the past by van den Oord et al. (2018) using Contrastive Predictive Coding (CPC), the results were mixed with marginal gains in a few DMLab (Espeholt et al.”
Representation Learning via Invariant Causal Mechanisms (2020, unknown): “However, (Tschannen et al., 2019) has shown that performance on downstream tasks may be more tightly correlated with the choice of encoder architecture than the achieved MI bound, highlighting issues with the MI theory of contrastive learning.”
Self-supervised Learning from a Multi-view Perspective (2020, methodology): “This result has also been observed in prior work (Tschannen et al., 2019) and InfoMin (Tian et al., 2020), which claim the representations with maximal mutual information may not have the best performance.”
Adversarial Graph Augmentation to Improve Graph Contrastive Learning (2021, methodology): “Tschannen et al. have shown that for image classiﬁcation, representations capturing the information that is entirely irrelevant to the image labels are also able to maximize the mutual information but such representations are deﬁnitely not useful for image classiﬁcation [40].”
Graph Clustering with Graph Neural Networks (2020, background): “Unsupervised training of GNNs is commonly done via maximizing mutual information [3, 26, 53, 57] in a self-supervised fashion.”