More Bang for Your Buck: Natural Perturbation for Robust Question Answering

By Daniel Khashabi, Tushar Khot, Ashish Sabharwal. 2020

75 citations recorded by the source corpus.

While recent models have achieved human-level scores on many NLP datasets, we observe that they are considerably sensitive to small changes in input. As an alternative to the standard approach of addressing this issue by constructing training sets of completely new examples, we propose doing so via minimal perturbation of examples. Specifically, our approach involves first collecting a set of seed examples and then applying human-driven natural perturbations (as opposed to rule-based machine perturbations), which often change the gold label as well. Local perturbations have the advantage of being relatively easier (and hence cheaper) to create than writing out completely new examples. To evaluate the impact of this phenomenon, we consider a recent question-answering dataset (BoolQ) and study the benefit of our approach as a function of the perturbation cost ratio, the relative cost of perturbing an existing question vs. creating a new one from scratch. We find that when natural perturbations are moderately cheaper to create, it is more effective to train models using them: such models exhibit higher robustness and better generalization, while retaining performance on the original BoolQ dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “The YN datasets we use are BoolQ (Clark et al., 2019a) naturally-perturbed version of this dataset, BoolQ-NP (Khashabi et al., 2020), and the binary (yes/no) subset of MultiRC (Khashabi et al., 2018).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “,, m = n may decrease vocabulary diversity or induce additional data bias, echoing (Khashabi et al., 2020)).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Selsam et al. (2019), Tafjord et al. (2019), Lin et al. (2019), and Khashabi et al. (2020) designed their data collection process to include contrastive examples.”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, methodology): “…et al., 2020) 91.0 ERNIE3,10B (Fedus et al., 2021) 58.9 Masque (Nishida et al., 2019) 90.7 ALBERT, 31M (Jiang et al., 2020) 86.0 UnifiedQA-ft,11B (Khashabi et al., 2020) 86.4 UnifiedQA-ft,11B (Khashabi et al., 2020) 75.0 UnifiedQA-ft,11B (Khashabi et al., 2020) 95.0 UnifiedQA-ft,11B (Khashabi et…”
Perturbation Augmentation for Fairer NLP (2022, background): “Counterfactual data augmentation approaches are heavily inspired by recent work in the domain of robustness (Ribeiro et al., 2019; Morris et al., 2020; Khashabi et al., 2020; Balashankar et al., 2021; Liu et al., 2021; Zhang et al., 2021; Sen et al., 2021; Papakipos and Bitton, 2022).”
Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data (2020, methodology): “We follow a similar setup to Khashabi et al. (2020) and use English NLI data to test whether counterfactually-augmented training data yields models that generalize better to out-of-domain data and are more robust to challenge examples.”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “We collect the existing Boolean QA datasets: BoolQ (Clark et al., 2019a), BoolQ-NP (Khashabi et al., 2020), BoolQ-CS (Gardner et al., 2020), StrategyQA (Geva et al., 2021), and extract the questions in MultiRC dataset (Khashabi et al., 2018) that can be answered with Yes/No as the data for generic…”
Can NLI Models Verify QA Systems' Predictions? (2021, methodology): “Models focusing on robustness and generalizability have been proposed in recent years: Wang and Bansal (2018); Khashabi et al. (2020); Liu et al. (2020) use perturbation based methods and adversarial training; Lewis and Fan (2018) propose generative QA to prevent the model from overﬁtting to simple…”