Flow of Information in Feed-Forward Deep Neural Networks
By Pejman Khadivi, Ravi Tandon, Naren Ramakrishnan. 2016
20 citations recorded by the source corpus.
Feed-forward deep neural networks have been used extensively in various machine learning applications. Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment. In this paper, we use an information theoretic approach to study the flow of information in a neural network and to determine how entropy of information changes between consecutive layers. Moreover, using the Information Bottleneck principle, we develop a constrained optimization problem that can be used in the training process of a deep neural network. Furthermore, we determine a lower bound for the level of data representation that can be achieved in a deep neural network with an acceptable level of distortion.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Jihong Park
- Sumudu Samarakoon
- Merouane Debbah
- Shrikanth S. Narayanan
- Anis Elgabli
- H. Vincent Poor
- Seong-Lyun Kim
Citations in the corpus
- Wireless Network Intelligence at the Edge (2018, methodology): “When the bottlenecks are discrete random variables, the flow dynamics are described by the discrete Shannon entropy [155].”
- Flow of Renyi information in deep neural networks (2016, background): “[4] characterized the discrete entropy change between consecutive layers based on the IB principle.”