Adaptive Estimators Show Information Compression in Deep Neural Networks
By Ivan Chelombiev, Conor J. Houghton, Cian O’Donnell. 2019
37 citations recorded by the source corpus.
To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions. In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression. In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions. Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions. With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions. We also find that there is a large amount of variation in compression between different network initializations. Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting. Finally, we show that only compression of the last layer is positively correlated with generalization.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alessandro Achille
- Stefano Soatto
- Giovanni Paolini
- Aditya Golatkar
- Avinash Ravichandran
- Greg Ver Steeg
- Sergey Levine
- Yoshua Bengio
Citations in the corpus
- Where is the Information in a Deep Neural Network? (2019, background): “…complete, are a deterministic function of the input: [16] argue that DNNs work by compressing the inputs, but the experimental setup has been challenged [15] and no consensus has been reached [6], as using a deterministic map to compress a continuous random variable presents technical challenges.”