Improving the Robustness of Question Answering Systems to Question Paraphrasing
By Wee Chung Gan, Hwee Tou Ng. 2019
147 citations recorded by the source corpus.
Despite the advancement of question answering (QA) systems and rapid improvements on held-out test sets, their generalizability is a topic of concern. We explore the robustness of QA models to question paraphrasing by creating two test sets consisting of paraphrased SQuAD questions. Paraphrased questions from the first test set are very similar to the original questions designed to test QA models’ over-sensitivity, while questions from the second test set are paraphrased using context words near an incorrect answer candidate in an attempt to confuse QA models. We show that both paraphrased test sets lead to significant decrease in performance on multiple state-of-the-art QA models. Using a neural paraphrasing model trained to generate multiple paraphrased questions for a given source question and a set of paraphrase suggestions, we propose a data augmentation approach that requires no human intervention to re-train the models for improved robustness to question paraphrasing.
Subfields: Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Yoav Goldberg
- Mohit Bansal
- Yonatan Belinkov
- Graham Neubig
- Sebastian Riedel
- Hinrich Schütze
- Yejin Choi
- Luke S. Zettlemoyer
In Safety, Robustness & Interpretability
- Hinrich Schütze
- Yanai Elazar
- Shauli Ravfogel
- Yoav Goldberg
- Eduard H. Hovy
- Abhilasha Ravichander
- Sebastian Riedel
- Jonathan Berant
Citations in the corpus
- MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Finally, a large body of work suggests QA models are over-reliant on word-matching between question and context (Jia and Liang, 2017; Gan and Ng, 2019).”
- Measuring and Improving Consistency in Pretrained Language Models (2021, methodology): “However, the results perform on par with the baseline model, both on SQuAD and the paraphrase questions.”
- What Do Models Learn from Question Answering Datasets? (2020, background): “Experiments on SQuAD in particular have shown that SQuAD models are vulnerable to adversarial attacks (Jia and Liang, 2017; Wallace et al., 2019) and not robust to paraphrases (Ribeiro et al., 2018; Gan and Ng, 2019).”
- BERTese: Learning to Speak to BERT (2021, background): “Jiang et al. (2020) and Heinzerling and Inui (2020) then showed that in the zeroshot setting, small variations to such queries can lead to a drop in fact recall.”