Direct Validation of the Information Bottleneck Principle for Deep Nets

By Adar Elad, Doron Haviv, Yochai Blau, Tomer Michaeli. 2019

27 citations recorded by the source corpus.

The information bottleneck (IB) has been suggested as a fundamental principle governing performance in deep neural nets (DNNs). This idea sparked research on the information plane dynamics during training with the cross-entropy loss, and on using the IB of some "bottleneck" layer as a loss function. However, the claim that reaching the maximal value of the IB Lagrangian in each layer leads to optimal performance, was in fact never directly confirmed. In this paper, we propose a direct way of validating this hypothesis, using layer-by-layer training with the IB loss. In accordance with the original theory, we train each DNN layer explicitly with the IB objective (and without any classification loss), and freeze it before moving on to train the next layer. While mutual information (MI) is generally hard to estimate in high dimensions, we show that in the case of MI between DNN layers, this can be done quite accurately using a modification of the recently proposed mutual information neural estimator. Interestingly, we find that layer-by-layer training with the IB loss leads to accuracy which is on-par with end-to-end training with the cross entropy loss. This is, thus, the first direct experimental illustration of the link between the IB value in each layer, and a net's performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Simone Scardapane
2. Michele Scarpiniti
3. Aurelio Uncini
4. Enzo Baccarelli
5. Jiangpeng He
6. Feng Zhu
7. Laura Palagi

In Computer Vision and Pattern Recognition

1. Michele Scarpiniti
2. Enzo Baccarelli
3. Simone Scardapane
4. Aurelio Uncini
5. Feng Zhu
6. Jiangpeng He
7. Laura Palagi

In Scaling & Learning Theory

1. Simone Scardapane

In Statistical Learning Theory

1. Michele Scarpiniti
2. Aurelio Uncini
3. Enzo Baccarelli
4. Simone Scardapane

Citations in the corpus

• Why Should We Add Early Exits to Neural Networks? (2020, background): “Alternative supervised and unsupervised training strategies have also been proposed in a number of papers [19, 43].”