Estimating Information Flow in Neural Networks

By Ziv Goldfeld, Ewout van den Berg, Kristjan H. Greenewald, Igor Melnyk, Nam H. Nguyen, Brian Kingsbury, Yury Polyanskiy. 2018

32 citations recorded by the source corpus.

We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information $I(X;T)$ between the input $X$ and internal representations $T$ decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true $I(X;T)$ over these networks is provably either constant (discrete $X$) or infinite (continuous $X$). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which $I(X;T)$ is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models. By relating $I(X;T)$ in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the $T$ space. Finally, we return to the estimator of $I(X;T)$ employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Where is the Information in a Deep Neural Network? (2019, background): “A similar point of view is used later by [9], which estimate mutual information under the hypothesis of inputs with isotropic noise.”
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle (2018, background): “Thus, in light of the discussion in this work and in [14], [23], [29] along with the empirical evidence in [35], [36], we conclude that designing regularizers directly with the aim of instilling certain properties desirable for the intermediate representation L (such as discussed in Sec.”
Shredder: Learning Noise Distributions to Protect Inference Privacy (2019, methodology): “Mutual information has been widely used in the literature for both understanding the behavior of neural networks [26, 27,27–30], andalso toquantify information leakage inanonymity systems in the context of databases [31–33].”
Shredder: Learning Noise to Protect Privacy with Partial DNN Inference on the Edge (2019, background): “Mutual information has been widely used in the literature for both understanding the behaviour of neural networks [24, 25, 24, 26, 27, 28], and also to quantify information leakage in anonymity systems in the context of databases [29, 30, 31].”
A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability (2018, methodology): “[Goldfeld et al., 2018] suggested relating the mapping X 7→ Tk to the input and the output of a stochastic parameterised communication channel with parameters being DNN’s weights and biases.”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “Our theoretical results are also independent of the practical issue of estimating MI between neural network layers, an active area of recent research (Belghazi et al., 2018; Goldfeld et al., 2018; Dai et al., 2018; Gabrié et al., 2018), though our empirical experiments in Section 7 rely on the estim”