Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering

By Wenhan Xiong, Hong Wang, William Yang Wang. 2020

18 citations recorded by the source corpus.

Commonly used information retrieval methods such as TF-IDF in open-domain question answering (QA) systems are insufficient to capture deep semantic matching that goes beyond lexical overlaps. Some recent studies consider the retrieval process as maximum inner product search (MIPS) using dense question and paragraph representations, achieving promising results on several information-seeking QA datasets. However, the pretraining of the dense vector representations is highly resource-demanding, e.g., requires a very large batch size and lots of training steps. In this work, we propose a sample-efficient method to pretrain the paragraph encoder. First, instead of using heuristically created pseudo question-paragraph pairs for pretraining, we use an existing pretrained sequence-to-sequence model to build a strong question generator that creates high-quality pretraining data. Second, we propose a simple progressive pretraining algorithm to ensure the existence of effective negative samples in each batch. Across three open-domain QA datasets, our method consistently outperforms a strong dense retrieval baseline that uses 6 times more computation for training. On two of the datasets, our method achieves more than 4-point absolute improvement in terms of answer exact match.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wen-tau Yih
2. Danqi Chen
3. Luke S. Zettlemoyer
4. Sebastian Riedel
5. Hannaneh Hajishirzi
6. Patrick Lewis
7. Christopher D. Manning
8. Sewon Min

In Computer Vision and Pattern Recognition

1. Wen-tau Yih
2. Danqi Chen
3. Sewon Min
4. Luke S. Zettlemoyer
5. Patrick Lewis
6. Hannaneh Hajishirzi
7. Christopher D. Manning
8. Mohit Bansal

In Retrieval-Augmented Generation

1. Wen-tau Yih
2. Danqi Chen
3. Patrick Lewis
4. Christopher D. Manning
5. Sewon Min
6. Barlas Oğuz
7. Sebastian Riedel
8. Gautier Izacard

In Information Retrieval & Search

1. Wen-tau Yih
2. Danqi Chen
3. Patrick Lewis
4. Sewon Min
5. Christopher D. Manning
6. Sebastian Riedel
7. Barlas Oğuz
8. Gautier Izacard

Citations in the corpus

• Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “t relying on additional pretraining or complex joint training schemes. DPR has also been used as an important module in very recent work. For instance, extending the idea of leveraging hard negatives,Xiong et al. (2020a) use the retrieval model trained in the previous iteration to discover new negat”
• You Only Need One Model for Open-domain Question Answering (2021, methodology): “TAS (Hofstätter et al., 2021) and Xiong et al. (2021b) used clustering of embeddings for the same.”