Where is the Information in a Deep Neural Network?

By Alessandro Achille, Giovanni Paolini, Stefano Soatto. 2019

97 citations recorded by the source corpus.

Whatever information a deep neural network has gleaned from training data is encoded in its weights. How this information affects the response of the network to future data remains largely an open question. Indeed, even defining and measuring information entails some subtleties, since a trained network is a deterministic map, so standard information measures can be degenerate. We measure information in a neural network via the optimal trade-off between accuracy of the response and complexity of the weights, measured by their coding length. Depending on the choice of code, the definition can reduce to standard measures such as Shannon Mutual Information and Fisher Information. However, the more general definition allows us to relate information to generalization and invariance, through a novel notion of effective information in the activations of a deep network. We establish a novel relation between the information in the weights and the effective information in the activations, and use this result to show that models with low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs. These relations hinge not only on the architecture of the model, but also on how it is trained, highlighting the complex inter-dependency between the class of functions implemented by deep neural networks, the loss function used for training them from finite data, and the inductive bias implicit in the optimization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Information Theory

Citations in the corpus

Training Neural Networks with Fixed Sparse Masks (2021, background): “…learning, e.g. as a measure of parameter importance [21], as a preconditioner in gradient descent [4, 37, 34], as a way to measure the amount of “information” in each parameter of a neural network [1], or as a way to decide which parameters to prune when performing model compression [43, 10, 47].”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “, information theory [39], is a very interesting venue for future work.”
Mixed-Privacy Forgetting in Deep Networks (2020, background): “They also bound the amount of remaining information in the weights [3] after scrubbing.”
Generalized Variational Continual Learning (2020, background): “From an information-theoretic perspective, KL-reweighted ELBOs have also studied as compression (Achille et al., 2020).”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, background): “Forgetting is also intrinsically connected with information: Forgetting may also be seen as minimizing an upper-bound on the amount of information that the weights contain about Df [3] and that an attacker may extract about that particular cohort Df using some readout function f .”
Merging Models with Fisher-Weighted Averaging (2021, background): “The knowledge contained within the parameters of a network will necessarily be greater than the knowledge contained in its output [2].”
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting (2020, background): “Achille et al. (2019) argued that the information in the weights controls the PAC-Bayesian bound.”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “Achille and Soatto (2019) also obtained the closely related Proposition 3.6 that describes the mean escape time under the isotropic gradient noise in terms of a free energy that depends on the Fisher Information.”
Machine unlearning: linear filtration for logit-based classifiers (2020, background): “[28]), hence unlearning as defined above may currently be out of reach.”
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations (2020, background): “1 Indeed, it is even difficult to define, as the quantities at play, Df and x, are fixed values, but that problem has been addressed by [2] and we do not consider it here.”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “As a consequence of their extreme sensitivity, artificial neural network models have high curvature decision boundaries [71], [72], whereas decision tree-based algorithms can learn hyperplane-like boundaries.”