Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering

By Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, Jimmy J. Lin. 2019

70 citations recorded by the source corpus.

Recently, a simple combination of passage retrieval using off-the-shelf IR techniques and a BERT reader was found to be very effective for question answering directly on Wikipedia, yielding a large improvement over the previous state of the art on a standard benchmark dataset. In this paper, we present a data augmentation technique using distant supervision that exploits positive as well as negative examples. We apply a stage-wise approach to fine tuning BERT on multiple datasets, starting with data that is "furthest" from the test data and ending with the "closest". Experimental results show large gains in effectiveness over previous approaches on English QA datasets, and we establish new baselines on two recent Chinese QA datasets.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sebastian Riedel
2. Wen-tau Yih
3. Jimmy J. Lin
4. Hannaneh Hajishirzi
5. Danqi Chen
6. Luke S. Zettlemoyer
7. Mohit Bansal
8. Patrick Lewis

In Information Retrieval & Search

1. Wen-tau Yih
2. Jimmy J. Lin
3. Sebastian Riedel
4. Danqi Chen
5. Patrick Lewis
6. Wei Yang
7. Sewon Min
8. Barlas Oğuz

In Natural Language Processing

1. Sebastian Riedel
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Hannaneh Hajishirzi
5. Wen-tau Yih
6. Danqi Chen
7. Jianfeng Gao
8. Patrick Lewis

Citations in the corpus

• Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “…open-domain QA to machine reading is a very reasonable strategy, a huge performance degradation is often observed in practice (Chen et al., 2017; Yang et al., 2019) 2 , which suggests that the retriever needs to be enhanced and the reader also requires to be adjusted for multi-context input.”
• PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “Following recent best practices(Alberti et al., 2019; Yang et al., 2019), we then fine-tune on the training QA-pairs only, using validation Exact Match score for early stopping (Rajpurkar et al., 2018).”
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, methodology): “This direction has been further explored at a larger scale with efficient Transformer architectures (Radford et al., 2019; Devlin et al., 2019; Yang et al., 2019c).”
• What Do Models Learn from Question Answering Datasets? (2020, neutral): “Reference Ours SQuAD 76.3 (Liu et al., 2019) 75.6 TriviaQA 56.3 (Yang et al., 2019) 58.7 NQ 52.7 (Alberti et al., 2019) 73.5 QuAC 54.4 (Qu et al., 2019) 33.3 NewsQA 66.8 (Takahashi et al., 2019) 60.1”
• Learning with Limited Data for Multilingual Reading Comprehension (2019, methodology): “For negative examples, we generate synthetic noises, similarly to translation/generation errors, by adopting a naive but effective method to change words or sentences in a positive example (Levy et al., 2017; Zhang et al., 2019; Yang et al., 2019).”
• Distant Supervision for Multi-Stage Fine-Tuning in Retrieval-Based Question Answering (2020, methodology): “When our distant supervision techniques were first proposed [30], we reported the best known score on SQuAD that we were aware of.”