CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
By Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin. 2020
541 citations recorded by the source corpus.
Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce an accelerated MI minimization training scheme, which bridges MI minimization with negative sampling. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, further demonstrate the effectiveness of the proposed method.
Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Representation & Self-Supervised Learning
In Information Theory
Citations in the corpus
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, neutral): “The “InfoLOOB” objective is called “Leave one out upper bound” in Poole et al. (2019) and “L1Out” in Cheng et al. (2020).”
- Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “However, the computation of MI values is intractable [44,18], since the form of joint probability distribution p ( a , b ) in Eq.”
- InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “MINE [51] and InfoNCE [15] compute a lower bound as the MI estimators while CLUB [52] computes an upper bound as the MI estimator.”
- Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021, methodology): “Following Cheng et al. (2020), we formulate q ( y | x ) as a multivariate Gaussian distributions q θ ( y | x ) = N ( y | µ θ 1 ( x ) , σ 2 θ 2 ( x ) I ) , with two neural networks parameterized by θ 1 and θ 2 to predict the mean and variance, respectively.”
- InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective (2020, methodology): “, Maximum Log Likelihood) • The second term uses CLUB [1] to calculate the upper bound • The last term uses InfoNCE as the lower bound Task objective Information Bottleneck Regularizer Local Anchored Feature Regularizer”
- Learning to Diversify for Single Domain Generalization (2021, background): “The upper bound of MI defined in [7] is:”
- Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, methodology): “In our work we use contrastive log-ratio upper bound (CLUB) [20] of the MI to approximate it.”
- Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, methodology): “Then we propose a system consisting of four components: (1) Random resampling [23] operation along the time dimension is performed on pitch and content to remove rhythm information; (2) A common classifier is used for timbre to extract features related to speaker identity, and a gradient reversal la”
- Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “Estimation of mutual information (MI) is central to many scientific investigations and engineering problems [36–38], thus motivating a plethora of practical procedures [39–46].”
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, background): “The variational contrastive log-ratio upper bound (vCLUB) (Cheng et al., 2020) of mutual information is defined by: I(X;Y ) = Ep(X,Y )[log qθ(Y |X)]− Ep(X)p(Y )[log qθ(Y |X)] (2) where qθ(Y |X) is the variational distribution to approximate the ground-truth probability P (Y |X) and can be parameteri”