CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

By Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin. 2020

541 citations recorded by the source corpus.

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce an accelerated MI minimization training scheme, which bridges MI minimization with negative sampling. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, further demonstrate the effectiveness of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Dacheng Tao
  3. Luc Van Gool
  4. Yu Cheng
  5. Xiaogang Wang
  6. Jianfeng Gao
  7. Lawrence Carin
  8. Jingjing Liu

In Representation & Self-Supervised Learning

  1. Zhe Gan
  2. Jingjing Liu
  3. Lawrence Carin
  4. Yu Cheng
  5. Ruihong Qiu
  6. Bo Li
  7. Wei Han
  8. Dong Yu

In Information Theory

  1. Wei Han
  2. Zhe Gan
  3. Lawrence Carin
  4. Soujanya Poria
  5. Hui Chen
  6. Shuyang Gao

Citations in the corpus