Deep learning and the information bottleneck principle
By Naftali Tishby, Noga Zaslavsky. 2015
2,084 citations recorded by the source corpus.
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Statistical and Nonlinear Physics
- Yoshua Bengio
- Stefano Ermon
- Aaron C. Courville
- Sergey Levine
- Jiaming Song
- Ben Poole
- Chelsea Finn
- Yann LeCun
In Scaling & Learning Theory
- Yoshua Bengio
- Geoffrey E. Hinton
- Stefano Soatto
- Michael W. Mahoney
- Alessandro Achille
- Andrew M. Saxe
- Tengyu Ma
- Charles H. Martin
Citations in the corpus
- The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “Tishby and Zaslavsky (2015) state that “the goal of any supervised learning is to capture and efficiently represent the relevant information in the input variable about the output-label variable” and hence the representations undergo transformations which aim to encode as much information about the…”
- GAIT: A Geometric Approach to Information Theory (2019, methodology): “We hope these methods can prove fruitful in extending frameworks such as the information bottleneck for representation learning (Tishby and Zaslavsky, 2015), similarity-sensitive cross entropy objectives in the spirit of loss-calibrated decision theory (Lacoste-Julien et al.”
- Wireless Network Intelligence at the Edge (2018, background): “b) Information bottleneck principle: The bias–variance tradeoff is well observed in the NN training process under information bottleneck (IB) [88], [89].”
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “[97], [98].”
- Opening the Black Box of Deep Neural Networks via Information (2017, background): “As proposed by Tishby & Zaslavsky (2015), we study the information paths of DNNs in the information plane .”
- TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2018, methodology): “Information bottleneck theory [10] has also gained popularity as a method for explaining the generalization and learning capabilities of deep learning based models.”
- Critical Learning Periods in Deep Neural Networks (2017, background): “While some connections have been established between these three aspects (16, 17), this remains a largely unexplored field of research.”
- Excessive Invariance Causes Adversarial Vulnerability (2018, background): “Finally, the information-theoretic view has gained recent interest due to the information bottleneck (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Alemi et al., 2017) and usage in generative modelling (Chen et al., 2016; Hjelm et al., 2018).”
- i-RevNet: Deep Invertible Networks (2018, background): “Tishby & Zaslavsky (2015) introduces the information bottleneck principle which states that an optimal representation must reduce the mutual information between an input and its representation to reduce as much uninformative variability as possible.”
- Where is the Information in a Deep Neural Network? (2019, methodology): “The Information Lagrangian is in the general form of an Information Bottleneck (IB) [17], but is fundamentally different from the IB used in most prior work in deep learning [18], which refers to the activations, rather than the weights.”
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “We explore in this appendix the connection between B AR - LOW T WINS ’ loss function and the Information Bottleneck (IB) principle (Tishby & Zaslavsky, 2015; Tishby et al., 2000).”
- On Variational Bounds of Mutual Information (2019, methodology): “Mutual information estimators are used in computational neuroscience (Palmer et al., 2015), Bayesian optimal experimental design (Ryan et al., 2016; Foster et al., 2018), understanding neural networks (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Gabri´e et al., 2018), and more.”
- Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “This achievement may help us understand to what degree discarding information is crucial to deep learning’s success (Tishby and Zaslavsky, 2015).”
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “, 2020), we hypothesize that the information bottleneck principle (Tishby and Zaslavsky, 2015) tends to transform the cross-lingual structural similarity into isomorphic representations, which has similar bridge effects as the anchor points.”
- Mutual Information Neural Estimation (2018, background): “IB was recently covered in the context of deep learning (Tishby & Zaslavsky, 2015), and as such can be seen as a process to construct an approximation of the minimally sufficient statistics of the data.”
- The Description Length of Deep Learning models (2018, background): “…in the weights: there is no mutual information between inputs and outputs. how the hidden representations in a neural network compress the inputs while preserving the mutual information between inputs and outputs (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2017).”
- Taming VAEs (2018, background): “While a body of work on information bottleneck has primarily focused on tools to analyze models [31, 32, 33], it has also been shown that VAEs with various information bottleneck constraints can trade off reconstruction accuracy for better-factorized latent representations [30, 34, 35], a highly…”
- Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “Efforts to develop a theoretical framework for representation learning include Tishby and Zaslavsky (2015) and Shwartz-Ziv and Tishby (2017), who consider representations as stochastic functions that approximate minimal sufficient statistics, different from Bruna and Mallat (2011) who construct…”