Distant Supervision for Multi-Stage Fine-Tuning in Retrieval-Based Question Answering

By Yuqing Xie, Wei Yang, Luchen Tan, Kun Xiong, Nicholas Jing Yuan, Baoxing Huai, Ming Li, Jimmy J. Lin. 2020

23 citations recorded by the source corpus.

We tackle the problem of question answering directly on a large document collection, combining simple “bag of words” passage retrieval with a BERT-based reader for extracting answer spans. In the context of this architecture, we present a data augmentation technique using distant supervision to automatically annotate paragraphs as either positive or negative examples to supplement existing training data, which are then used together to fine-tune BERT. We explore a number of details that are critical to achieving high accuracy in this setup: the proper sequencing of different datasets during fine-tuning, the balance between “difficult” vs. “easy” examples, and different approaches to gathering negative examples. Experimental results show that, with the appropriate settings, we can achieve large gains in effectiveness on two English and two Chinese QA datasets. We are able to achieve results at or near the state of the art without any modeling advances, which once again affirms the cliché “there’s no data like more data”.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures (2020, methodology): “[17]: keyword search with BM25 followed by a BERT reader to extract the exact answer spans.”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “However, these PLMs may not perform optimally when applied to specific tasks like sentiment analysis, question answering, or translation due to a lack of appropriate domain knowledge [69], [70], [71].”
A Replication Study of Dense Passage Retriever (2021, result): “Similar analyses in previous work with BM25 retrieval have shown that end-to-end QA effectiveness increases with increasing k (Yang et al., 2019; Xie et al., 2020); that is, the reader does not appear to be “confused” by the non-relevant material.”
Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering (2020, methodology): “1 dataset is based on a few hundreds of Wikipedia articles, hence the measured eQA performance on the dev set can be partly due to overspecialization on a few topics (Xie et al. 2020).”