NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks

By Fahim Dalvi, Avery Nortonsmith, Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, James R. Glass. 2018

54 citations recorded by the source corpus.

We present a toolkit to facilitate the interpretation and understanding of neural network models. The toolkit provides several methods to identify salient neurons with respect to the model itself or an external task. A user can visualize selected neurons, ablate them to measure their effect on the model accuracy, and manipulate them to control the behavior of the model at the test time. Such an analysis has a potential to serve as a springboard in various research directions, such as understanding the model, better architectural choices, model distillation and controlling data biases. The toolkit is available for download.1

Subfields: Artificial Intelligence, Biophysics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Safety, Robustness & Interpretability

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “NeuroX (Dalvi et al., 2019b) is a tool for ﬁnding and analyzing individual neurons, focusing on machine translation.”
Interpretability and Analysis in Neural NLP (2020, methodology): “We will also present various interactive visualization methods such as neuron activations (Karpathy et al., 2015; Dalvi et al., 2019), attention mechanisms (Bah-danau et al., 2014; Strobelt et al., 2018), and saliency measures (Li et al., 2016; Murdoch et al., 2018; Arras et al., 2017), including a…”
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation (2022, methodology): “Probe-X (Ravishankar et al., 2019b) has expanded SentEval setup with 5 additional languages, while NeuroX framework (Dalvi et al., 2019) also introduced novelty, but proposed to enrich the methodology to allow for cross-model analysis of the results, supporting neuron-level inspection.”
Compositional Explanations of Neurons (2020, methodology): “Prior work in neural network interpretability has found that neurons in models trained for a variety of tasks learn human-interpretable concepts, e.g. faces or parts-of-speech, often without explicit supervision [5, 10, 11, 27].”
On the Linguistic Representational Power of Neural Machine Translation Models (2019, methodology): “An open-source implementation of our analysis code is available through the NeuroX toolkit (Dalvi et al. 2019b).”