Opening the Black Box of Deep Neural Networks via Information

By Ravid Shwartz-Ziv, Naftali Tishby. 2017

1,641 citations recorded by the source corpus.

Despite their great success, there is still no comprehensive theoretical understanding of learning with Deep Neural Networks (DNNs) or their inner organization. Previous work proposed to analyze DNNs in the \textit{Information Plane}; i.e., the plane of the Mutual Information values that each layer preserves on the input and output variables. They suggested that the goal of the network is to optimize the Information Bottleneck (IB) tradeoff between compression and prediction, successively, for each layer. In this work we follow up on this idea and demonstrate the effectiveness of the Information-Plane visualization of DNNs. Our main results are: (i) most of the training epochs in standard DL are spent on {\emph compression} of the input to efficient representation and not on fitting the training labels. (ii) The representation compression phase begins when the training errors becomes small and the Stochastic Gradient Decent (SGD) epochs change from a fast drift to smaller training error into a stochastic relaxation, or random diffusion, constrained by the training error value. (iii) The converged layers lie on or very close to the Information Bottleneck (IB) theoretical bound, and the maps from the input to any hidden layer and from this hidden layer to the output satisfy the IB self-consistent equations. This generalization through noise mechanism is unique to Deep Neural Networks and absent in one layer networks. (iv) The training time is dramatically reduced when adding more hidden layers. Thus the main advantage of the hidden layers is computational. This can be explained by the reduced relaxation time, as this it scales super-linearly (exponentially for simple diffusion) with the information compression from the previous layer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “[SZT17] identiﬁed a signal-dominated and noise-dominated phase of training.”
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “To get such distribution, in the original works (Shwartz-Ziv and Tishby, 2017), the authors binned the neuron’s arctan activations.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, positive): “This result confirms one of the findings of Tishby and Schwartz-Ziv [43] and Zhang et al. [57], who argue that neural networks first learn to minimize the loss on the training data by memorizing it.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[74] Ravid Shwartz-Ziv and Naftali Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.”
Why Should We Add Early Exits to Neural Networks? (2020, background): “4 While the IB principle has received ample empirical validation [55, 59], it is common to evaluate and/or apply it only to a single layer of the network.”
Are Sixteen Heads Really Better than One? (2019, methodology): “The two phases of training are reminiscent of the analysis by Shwartz-Ziv and Tishby (2017), according to which the training of neural networks decomposes into an “emiprical risk minimization” phase, where the model maximizes the mutual information of its intermediate representations with the…”
Differentially Private Federated Learning: A Client Level Perspective (2017, background): “3, Shwartz-Ziv and Tishby Shwartz-Ziv & Tishby (2017) suggest, we can distinguish two different phases of training: label fitting and data fitting phase.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “re: [14], [42], [84], [93] N.A. N.A. # Also listed elsewhere: [14], [59], [84] etc N.A. N.A. Optimization CNN with separable model [141] 7 X Others Information theoretic: Information Bottleneck [97], [98] 7 X Database of methods v.s. interpretability [10] N.A. N.A. Data Driven Persp. Case-Based Reas”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…Mooney, 2017; Reed, Akata, Lee, and Schiele, 2016; Sadeghi, Kumar Divvala, and Farhadi, 2015; Samek et al., 2017; Schank, 2013; Schmidhuber, 2015; Shwartz-Ziv and Tishby, 2017; Simonyan, Vedaldi, and Zisserman, 2013; Springenberg, Dosovitskiy, Brox, and Riedmiller, 2014; Sturm, Lapuschkinb,…”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “ture interpretability. 2 Related work Network interpretability: Recent studies of the interpretability of visual network are reviewed in [46]. Instead of analyzing network features from a global view [36, 26, 23], Bau et al. [2] deﬁned six kinds of semantics for middle-layer feature maps of a CNN, i”
A Survey of Methods for Explaining Black Box Models (2018, background): “in Reference [108] proposes f that shows the effectiveness of the Information Plane visualization of DNNs r , by highlighting that the empirical error minimization of each stochastic gradient descent phase epoch is always followed by a slow representation compression.”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “Many studies have been proposed to explain and interpret deep neural networks [70], [84], [119], [122].”
Visual Feature Attribution Using Wasserstein GANs (2017, background): “This behaviour is consistent with the network compressing away less predictive features discussed earlier [50].”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, background): “It can be shown that from layer to layer, networks compress information and enhance the signal-to-noise ratio (see [109,110]).”
Critical Learning Periods in Deep Neural Networks (2017, background): “A different two-phase interpretation of SGD has also been proposed in (17): In the authors’ analysis, the network first learns to encode the task in the network’s activations, while in a second phase it minimizes the quantity of information, while keeping enough of it to maintain good task performan”