Formal Limitations on the Measurement of Mutual Information

By David A. McAllester, Karl Stratos. 2018

337 citations recorded by the source corpus.

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistics and Probability, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Ruslan Salakhutdinov
  5. Jianfeng Gao
  6. Quoc V. Le
  7. Percy Liang
  8. Pieter Abbeel

In Computational Theory and Mathematics

  1. Sergey Levine
  2. Shixiang Shane Gu
  3. George Tucker
  4. Jimmy Ba
  5. Timothy P. Lillicrap
  6. Mohammad Norouzi
  7. Stefano Ermon
  8. Ofir Nachum

In Statistics and Probability

  1. Arash Vahdat
  2. Stefano Ermon
  3. Ben Poole
  4. Shixiang Shane Gu
  5. Danijar Hafner
  6. Sherjil Ozair
  7. Aäron van den Oord
  8. Timothy P. Lillicrap

In Information Theory

  1. Stefano Ermon
  2. Jiaming Song
  3. Alexander A. Alemi
  4. William Buchwalter
  5. Tiago Pimentel
  6. Benjamin Bloem-Reddy
  7. Michael Tschannen
  8. Philip Bachman

Citations in the corpus