Deep learning and the information bottleneck principle

By Naftali Tishby, Noga Zaslavsky. 2015

2,084 citations recorded by the source corpus.

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Sergey Levine
  4. Jianfeng Gao
  5. Percy Liang
  6. Quoc V. Le
  7. Ruslan Salakhutdinov
  8. Luke S. Zettlemoyer

In Statistical and Nonlinear Physics

  1. Yoshua Bengio
  2. Stefano Ermon
  3. Aaron C. Courville
  4. Sergey Levine
  5. Jiaming Song
  6. Ben Poole
  7. Chelsea Finn
  8. Yann LeCun

In Scaling & Learning Theory

  1. Yoshua Bengio
  2. Geoffrey E. Hinton
  3. Stefano Soatto
  4. Michael W. Mahoney
  5. Alessandro Achille
  6. Andrew M. Saxe
  7. Tengyu Ma
  8. Charles H. Martin

Citations in the corpus