Did the Model Understand the Question?

By Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, Kedar Dhamdhere. 2018

206 citations recorded by the source corpus.

We analyze state-of-the-art deep learning models for three tasks: question answering on (1) images, (2) tables, and (3) passages of text. Using the notion of “attribution” (word importance), we find that these deep networks often ignore important question terms. Leveraging such behavior, we perturb questions to craft a variety of adversarial examples. Our strongest attacks drop the accuracy of a visual question answering model from 61.1% to 19%, and that of a tabular question answering model from 33.5% to 3.3%. Additionally, we show how attributions can strengthen attacks proposed by Jia and Liang (2017) on paragraph comprehension models. Our results demonstrate that attributions can augment standard measures of accuracy and empower investigation of model performance. When a model is accurate but for the wrong reasons, attributions can surface erroneous logic in the model that indicates inadequacies in the test data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Model Cards for Model Reporting (2018, background): “Additional details may include, for example, interpretability approaches, such as saliency maps, TCAV [33], and Path-Integrated Gradients [38, 43]); stakeholder-relevant explanations (e.”
Counterfactual Fairness in Text Classification through Robustness (2018, methodology): “When applied to the text setting, such adversarial examples can be generated by a variety of editing methods, including through translation (Ribeiro, Singh, and Guestrin 2018), attributions (Mudrakarta et al. 2018), and autoencoders (Zhao, Dua, and Singh 2017) (Hu et al.”
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets (2019, background): “These challenges have been used as evidence that current systems are brittle (Belinkov and Bisk, 2018; Mudrakarta et al., 2018; Zhao et al., 2018; Glockner et al., 2018; Ebrahimi et al., 2018; Ribeiro et al., 2018, 1 Often referred to as “adversarial datasets” or “attacks”.”
Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, background): “, by adding distracting sentences to the context in MC (Jia and Liang, 2017), or feeding partial questions or wrong images into models (Agrawal et al., 2016; Mudrakarta et al., 2018; Feng et al., 2018).”
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering (2018, background): “Other minor perturbations such as typos (Belinkov and Bisk, 2018), adding distractor sentences (Jia and Liang, 2017; Mudrakarta et al., 2018), or character replacements (Ebrahimi et al., 2018) preserve meaning while degrading model performance.”
What Do Models Learn from Question Answering Datasets? (2020, background): “…answering has found that state-of-the-art models can get good performance on incomplete input (Agrawal et al., 2016; Sugawara et al., 2018; Niven and Kao, 2019), under-rely on important words, (Mudrakarta et al., 2018), and over-rely on simple heuristics (Weissenborn et al., 2017; Ko et al., 2020).”
Learning Credible Deep Neural Networks with Rationale Regularization (2019, background): “For instance, in question answering (QA) tasks, DNNs rely more on function words rather than pay attention to task-speciﬁc verbs, nouns and adjectives to make decisions [6], [7].”
Fine-grained Sentiment Analysis with Faithful Attention (2019, background): “Typically if the model is good enough, it should naturally rely on what the human relies on; however, this is not always the case, i.e. as discussed by Rajpurkar et al. (2018) and Mudrakarta et al. (2018).”
SG-Net: Syntax-Guided Machine Reading Comprehension (2019, background): “…cloze-style test (Hill et al., 2015; Hermann et al., 2015; Zhang et al., 2018) to span-based answer extraction from passage (Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017; Rajpurkar et al., 2018) and multi-choice style ones (Lai et al., 2017) where the two latter ones are our…”
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder (2020, background): “…are carried out by adding distracting phrases (Jia and Liang, 2017; Chan et al., 2018), editing the words and characters directly (Papernot et al., 2016; Alzantot et al., 2018; Ebrahimi et al., 2017) or paraphrasing sentences (Iyyer et al., 2018; Ribeiro et al., 2018; Mudrakarta et al., 2018).”
Techniques for interpretable machine learning (2018, methodology): “ist among training data. For instance, a post-hoc attribution approach has been utilized to analyze three question answering models and attributes the models' prediction to words in the question [24]. The attribution visualizations show that these models often ignore the important part of the q”
Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering (2021, background): “For instance, Ribeiro et al. (2018) generated adversaries using para-phrasing, while Mudrakarta et al. (2018) perturbed questions based on attribution.”
Backdoors in Neural Models of Source Code (2020, background): “These have been extensively studied in the domain of computer vision [35, 20, 27, 8, 9] and natural-language processing [28, 16, 43, 19].”
Complexity-Based Prompting for Multi-Step Reasoning (2022, background): “Existing evidence shows that the expressive neural models can take shortcuts during reasoning, relying on spurious correlations that inevitably exist in the training data (Mudrakarta et al., 2018; Sugawara et al., 2018; Lai et al., 2021).”
Explicit Bias Discovery in Visual Question Answering Models (2018, methodology): “We choose this model for two reasons : ﬁrst, its simplicity (in other words, an absence of “bells and whistles”) makes it a good test-bed for our method and has been used by other works that explore the behavior of VQA algorithms [30, 16].”
Incorporating Priors with Feature Attribution on Text Classification (2019, methodology): “Hence, interpretability methods tailored for text are quite sparse (Mudrakarta et al., 2018; Jia and Liang, 2017; Murdoch et al., 2018).”