Net2Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks

By Ruth C. Fong, Andrea Vedaldi. 2018

295 citations recorded by the source corpus.

In an effort to understand the meaning of the intermediate representations captured by deep networks, recent papers have tried to associate specific semantic concepts to individual neural network filter responses, where interesting correlations are often found, largely by focusing on extremal filter responses. In this paper, we show that this approach can favor easy-to-interpret cases that are not necessarily representative of the average behavior of a representation. A more realistic but harder-to-study hypothesis is that semantic representations are distributed, and thus filters must be studied in conjunction. In order to investigate this idea while enabling systematic visualization and quantification of multiple filter responses, we introduce the Net2Vec framework, in which semantic concepts are mapped to vectorial embeddings based on corresponding filter responses. By studying such embeddings, we are able to show that 1., in most cases, multiple filters are required to code for a concept, that 2., often filters are not concept specific and help encode multiple concepts, and that 3., compared to single filter activations, filter embeddings are able to better characterize the meaning of a representation and its relationship to other concepts.

Subfields: Artificial Intelligence, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

Citations in the corpus

Interpreting CNNs via Decision Trees (2018, background): “Fong and Vedaldi [11] analyzed how multiple filters jointly represented a specific semantic concept.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, background): “In contrast, the hidden layer representa-a tion of semantic concepts is a distributed pattern [9].”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, background): “As embedding for a leaf concept (an output classes) they suggest to take the weights connecting the penultimate layer to a class output, and as similarity measure for the categories they use dot-product (cf. Net2Vec and TCAV). dimension reduction i.e., sample points are projected to a sub-space; Exa”
Understanding Deep Networks via Extremal Perturbations and Smooth Masks (2019, methodology): “In contrast to other methods, which explicitly aim to find class-specific channels and/or directions at a global level [7, 10, 33], we are able to similarly do so “for free” using only our per-instance channel attribution masks.”
A Survey on Neural Network Interpretability (2020, background): “Along the same lines, Fong and Vedaldi [67] investigate the embeddings of concepts over multiple kernels by constructing M with a combination of several kernels.”
Shape or Texture: Understanding Discriminative Features in CNNs (2021, background): “One downside of the increasing abstraction capabilities of deep CNNs is the lack of interpretability of the latent representations due to hidden layer activations coding semantic concepts in a distributed fashion (Fong & Vedaldi, 2018).”
Compositional Explanations of Neurons (2020, background): “Our work extends existing work on generating explanations for individual neurons in deep representations [4, 5, 10–12, 27] , in Complex explanations.”
Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition Under Occlusion (2020, background): “[15] shows that filter embeddings better characterize the meaning of a representation and its relationship to other concepts.”
Explaining Knowledge Distillation by Quantifying the Knowledge (2020, methodology): “Fong and Vedaldi [8] demonstrated that a DNN used multiple filters to represent a specific semantic concept.”
Biased Embeddings from Wild Data: Measuring, Understanding and Removing (2018, background): “However, there has been growing concern about potential biases in learning systems [1], [6] which can be difficult to analyse or query for explanations of their predictions, leading to an increasing number of studies investigating the way blackbox systems represent knowledge and make decisions [7], ”
Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs (2020, background): “While this suggests that the neurons provide a more interpretable basis compared to a random basis, [16] shows that the choice of basis is not the only challenge for interpretability of representations.”
Omni-Scale Feature Learning for Person Re-Identification (2019, background): “As feature channels encapsulate sophisticated correlations and can represent numerous visual concepts [7], it is advantageous to use channel-specific weights.”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Net2Vec [200] maps instead semantic concepts to corresponding individual DNN filter responses.”
Compositional Few-Shot Recognition with Primitive Discovery and Enhancing (2020, background): “Interpretability of deep networks has been studied in many aspects [5, 16, 59], showing that channels in the penultimate layer of deep networks correspond to some certain patterns patterns in the input images.”
Post-hoc Concept Bottleneck Models (2022, methodology): “In practice, densely annotated datasets (Caesar et al., 2018; Fong & Vedaldi, 2018) are used to collect points that are positive/negative for the concept.”