An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering

By Shayne Longpre, Yi Lu, Zhucheng Tu, Christopher DuBois. 2019

74 citations recorded by the source corpus.

To produce a domain-agnostic question answering model for the Machine Reading Question Answering (MRQA) 2019 Shared Task, we investigate the relative benefits of large pre-trained language models, various data sampling strategies, as well as query and context paraphrases generated by back-translation. We find a simple negative sampling technique to be particularly effective, even though it is typically used for datasets that include unanswerable questions, such as SQuAD 2.0. When applied in conjunction with per-domain sampling, our XLNet (Yang et al., 2019)-based submission achieved the second best Exact Match and F1 in the MRQA leaderboard competition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “On the Reddit and Amazon test sets, the best MRQA model in our testbed, Delphi (Longpre et al., 2019), achieves higher F1 scores than any SQuAD model and is substantially above the linear ﬁt.”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “little benefit for models like BERT and RoBERTa, which already achieve high performance on indomain text classification (Longpre et al., 2019b).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Thus, our dataset features a diverse collection of instructions (including human-written, template-based and LLM-generated), each paired with a human-written output.”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “(Longpre et al., 2019) The submission from Apple investigates the effects of pre-trained language models (BERT vs XLNet), various data sampling strategies, and data augmentation techniques via back-translation.”
Entity-Based Knowledge Conflicts in Question Answering (2021, methodology): “We use the RoBERTa (Liu et al., 2019) implementation from HuggingFace (Wolf et al., 2020) and hyperparameters from Longpre et al. (2019).”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “Unlike traditional DA techniques such as paraphrasing (Gangal et al., 2022) or back-translation (Longpre et al., 2019), which primarily aim at expanding the training dataset in a somewhat mechanical manner.”
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020, background): “For instance, Longpre et al. (2019) did not find standard BT useful for improving generalization of question answering models.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, unknown): “Rules [10], [7], [11] [100], [101] Machine Translation [42], [61], [10], [12], [60], [62], [64], [7], [19], [67], [102], [97], [103] [13], [59] [42], [58], [15], [104]”