Obtaining Faithful Interpretations from Compositional Neural Networks

By Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner. 2020

43 citations recorded by the source corpus.

Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture. However, prior work implicitly assumed that the structure of the network modules, describing the abstract reasoning process, provides a faithful explanation of the model’s reasoning; that is, that all modules perform their intended behaviour. In this work, we propose and conduct a systematic evaluation of the intermediate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multiple reasoning steps. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour. To remedy that, we train the model with auxiliary supervision and propose particular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Explainable Multi-hop Verbal Reasoning Through Internal Monologue (2021, methodology): “In contrast, we explicitly evaluate the performance of each component of EVR after training, achieving better faithfulness (Subramanian et al., 2020).”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “They learn all neural modules together simultaneously end-to-end, which fails to enable systematic generalization [3] and results in modules that are not faithful to their intended tasks [48], compromising the interpretability of the model.”
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language (2020, background): “This results in proofs that substantially exceed the earlier method’s accuracy, and also reﬂect the model’s internal decisions, rather than a post-hoc rationalization (i.e., is a “faithful” proof (Subramanian et al., 2020)).”
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021, background): “…PTLMs contain extensive world knowledge (Petroni et al., 2019; Roberts et al., 2020), they can be inconsistent in their answers to probing questions (Ettinger, 2020; Davison et al., 2019; Ravichander et al., 2020; Elazar et al., 2021; Subramanian et al., 2020), making their “world model” unclear.”
A Diagnostic Study of Explainability Techniques for Text Classification (2020, result): “This is in contrast to ML sub-ﬁelds such as image analysis, for which we only found one relevant dataset – 536 manually annotated object bounding boxes for Visual Question Answering (Subramanian et al., 2020).”
Enriching a Model's Notion of Belief using a Persistent Memory (2021, background): “, “Oxygen is colorless”); in other words, ensure complex answers were faithful to the BeliefBank (Subramanian et al., 2020).”
Post-hoc Interpretability for Neural NLP: A Survey (2021, methodology): “However, even intrinsically interpretable methods such as Attention and Neural Modular Networks have been shown to not reflect the model (Jain and Wallace, 2019; Subramanian et al., 2020).”
Modular Visual Question Answering via Code Generation (2023, background): “Also, obtaining interpretable module outputs from previous modular approaches is nontrivial (Subramanian et al., 2020), whereas in our approach the modules are frozen and thus interpretable.”