The information bottleneck method

By Naftali Tishby, Fernando C Pereira, William Bialek. 2000

4,490 citations recorded by the source corpus.

We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Management Science and Operations Research, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

  1. Medhi Bennis
  2. Merouane Debbah
  3. Yoshua Bengio
  4. Jihong Park
  5. Sergey Levine
  6. Andrew McCallum
  7. Daphne Koller
  8. Stefano Soatto

In Electrical and Electronic Engineering

  1. Sergey Levine
  2. Merouane Debbah
  3. Medhi Bennis
  4. Yoshua Bengio
  5. Chelsea Finn
  6. Ofir Nachum
  7. Pieter Abbeel
  8. George Tucker

In Management Science and Operations Research

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ruslan Salakhutdinov
  5. Ofir Nachum
  6. Sham M. Kakade
  7. George Tucker
  8. Mohammad Norouzi

In Information Theory

  1. Stefano Soatto
  2. Alessandro Achille
  3. Noam Slonim
  4. Greg Ver Steeg
  5. Adrian Weller
  6. Naftali Tishby
  7. Andrew McCallum
  8. Shrikanth S. Narayanan

Citations in the corpus