Understanding intermediate layers using linear classifier probes

By Guillaume Alain, Yoshua Bengio. 2016

1,640 citations recorded by the source corpus.

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself. This helps us better understand the roles and dynamics of the intermediate layers. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems. We apply this technique to the popular models Inception v3 and Resnet-50. Among other things, we observe experimentally that the linear separability of features increase monotonically along the depth of the model.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Designing and Interpreting Probes with Control Tasks (2019, background): “Early probing papers used linear functions (Shi et al., 2016; Ettinger et al., 2016; Alain and Bengio, 2016), which are still used (Bisazza and Tump, 2018; Liu et al., 2019), but multi-layer perceptron (MLP) probes are at least as popular (Belinkov et al., 2017; Conneau et al., 2018; Adi et al.,…”
Contrastive Representation Distillation (2019, methodology): “This is a common practice (Alain & Bengio, 2016; Zhang et al., 2017) to evaluate the quality of network representations; (b) fully ﬁnetuning : we fully ﬁnetune the ab network to obtain the best accuracy.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “A linear probe is used in [79] to extract information from each layer in a neural network.”
Sanity Checks for Saliency Maps (2018, neutral): “A number of results speak to the strength of randomly initialized models as classification priors [34, 35].”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “While prior work has considered fitting linear classifiers [51] or support vector machines [79], we chose the nearest neighbors because it explicits the relationship between predictions made by the model and its training data.”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “To further this point, Alain & Bengio (2016) ﬁnd that the features extracted from a randomly-initialized 3-hidden layer CNN on MNIST (the same architecture that is used in this work) lead to a 2 percent test error accuracy.”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, methodology): “Both (Bau et al., 2017) and (Alain & Bengio, 2016) show that meaningful directions can be efﬁciently learned via simple linear classiﬁers.”
Opening the Black Box of Deep Neural Networks via Information (2017, methodology): “Despite their great success, there is still no comprehensive understanding of the optimization process or the internal organization of DNNs, and they are often criticized for being used as mysterious ”black boxes” [e.g., Alain & Bengio (2016)].”
End-to-end differentiable learning of protein structure (2018, background): “It is also possible that the use of neural-network-probing techniques (Alain and Bengio, 2016; Koh and Liang, 2017; Nguyen et al., 2016; Shrikumar et al., 2017; Simonyan et al., 2013) with RGNs will provide new insight into the physical chemistry of folding and the types of intermediate structures t”
The emergence of number and syntax units in LSTM language models (2019, background): “Another research line studied linguistic processing in neural networks through ‘diagnos-tic classiﬁers’, that is, classiﬁers trained to predict a certain property from network activations (e.g., Gelderloos and Chrupała, 2016; Adi et al., 2017; Alain and Bengio, 2017; Hupkes et al., 2018).”
Investigating Transferability in Pretrained Language Models (2020, methodology): “…different layers of a neural network, typically by training a simple classifier to perform the task given only the layer representations as input (Alain and Bengio, 2016; Conneau et al., 2018; Hupkes et al., 2018; Liu et al., 2019; Tenney et al., 2019a,b; Goldberg, 2019; Hewitt †…”
Poisoning and Backdooring Contrastive Learning (2021, background): “As feature extractors for a second downstream classiﬁer (Alain & Bengio, 2016).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “This can be seen more as a distributed concept, though not simply a linear classifier probe (Alain and Bengio, 2017), as the Kohonen Neuron is found without access to the corresponding labels.”
Information-Theoretic Probing for Linguistic Structure (2020, background): “This contradicts the received wisdom that one should always select simple probes over more complex ones (Alain and Bengio, 2017; Liu et al., 2019; Hewitt and Manning, 2019).”
Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications (2017, neutral): “…performances across several ﬁelds, DL has been under scrutiny for its lack of transparency during the learning and testing processes (Alain and Bengio, 2016; Lou et al., 2012; Yosinski et al., 2015) For example, deep neural networks have been referred to as a “black box” in…”