Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
By Rana Ali Amjad, Bernhard C. Geiger. 2018
220 citations recorded by the source corpus.
In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, either the IB functional is infinite for almost all values of network parameters, making the optimization problem ill-posed, or it is piecewise constant, hence not admitting gradient-based optimization methods. Second, the invariance of the IB functional under bijections prevents it from capturing properties of the learned representation that are desirable for classification, such as robustness and simplicity. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results indicate limitations of the IB framework for the analysis of DNNs. We also note that rather than trying to repair the inherent problems in the IB functional, a better approach may be to design regularizers on latent representation enforcing the desired properties directly.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Representation & Self-Supervised Learning
In Information Theory
Citations in the corpus
- Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021, background): “From then on, numerous works studied and demonstrated the benefit of the MI-maximization principle (Bach-man et al., 2019; He et al., 2020; Amjad and Geiger, 2019).”
- The effectiveness of layer-by-layer training using the information bottleneck principle (2018, background): “One major concern highlighted in the recent works of Saxe et al. (2018) and Amjad & Geiger (2018), is that for deterministic DNNs with continuous inputs, the term I(X;Li) is always infinite so that the IB principle is in fact meaningless.”
- Caveats for information bottleneck in deterministic scenarios (2018, unknown): “Finally, there is the general notion that intermediate representations that are optimal in the IB sense correspond to “interesting” or “useful” compressions of input vectors (Amjad & Geiger, 2018).”
- Direct Validation of the Information Bottleneck Principle for Deep Nets (2019, background): “Strictly speaking, direct use of the IB functional for deterministic nets with continuous inputs is meaningless, as the term I(X;Li) in (1) is always infinite [12, 3].”