How Important Is a Neuron?

By Kedar Dhamdhere, Mukund Sundararajan, Qiqi Yan. 2018

159 citations recorded by the source corpus.

The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.

Subfields: Artificial Intelligence, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

Citations in the corpus

Model Fusion via Optimal Transport (2019, background): “5 , 2 } , LR decay epochs = { [20 , 40 , 60 , 80 , 100] , [10 , 30 , 50 , 70 , 90 , 110] , [30 , 60 , 90] } .”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “, 2017), layer conductance (Dhamdhere et al., 2019), integrated gradients (I.”
Captum: A unified and generic model interpretability library for PyTorch (2020, methodology): “This is especially useful for algorithms such as Integrated Gradients [17] and Layer Conductance [4] because they internally expand the inputs based on the number of integral approximation steps.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “Methodologically, sparse probing is situated within the broader category of probing methods [31, 32, 33, 34], which itself is just one paradigm among other localization techniques [35, 36, 37, 38, 39, 40].”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “Similar conservation properties have been noted in the network complexity [39], implicit regularization [40], and network interpretability [41, 42] literature with some highlighting the potential applications to pruning [9, 43].”
Influence-Directed Explanations for Deep Convolutional Networks (2018, background): “Concurrent work [20] has also suggested a slightly different approach; namely, measuring the input attribution that “flows through” a particular internal neuron.”
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020, methodology): “This view of a computation model is an extension of network decompositions from attribution methods using the natural concept of “layers” or “slices” (Dhamdhere et al., 2018; Leino et al., 2018; Bach et al., 2015).”
Towards Understanding Neural Machine Translation with Word Importance (2019, background): “The intermediate gradients have proven to be useful in interpreting deep learning models, such as NLP models (Mudrakarta et al., 2018; Dhamdhere et al., 2019) and computer vision models (Sel-varaju et al., 2017; Sundararajan et al., 2017).”