A Memory Efficient Baseline for Open Domain Question Answering

By Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, Edouard Grave. 2020

46 citations recorded by the source corpus.

Recently, retrieval systems based on dense representations have led to important improvements in open-domain question answering, and related tasks. While very effective, this approach is also memory intensive, as the dense vectors for the whole knowledge source need to be kept in memory. In this paper, we study how the memory footprint of dense retriever-reader systems can be reduced. We consider three strategies to reduce the index size: dimension reduction, vector quantization and passage filtering. We evaluate our approach on two question answering benchmarks: TriviaQA and NaturalQuestions, showing that it is possible to get competitive systems using less than 6Gb of memory.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sebastian Riedel
2. Hannaneh Hajishirzi
3. Luke S. Zettlemoyer
4. Wen-tau Yih
5. Mohit Bansal
6. Edouard Grave
7. Danqi Chen
8. Patrick Lewis

In Computer Vision and Pattern Recognition

1. Hannaneh Hajishirzi
2. Sebastian Riedel
3. Wen-tau Yih
4. Armand Joulin
5. Luc Van Gool
6. Edouard Grave
7. Patrick Lewis
8. Akari Asai

In Information Retrieval & Search

1. Sebastian Riedel
2. Wen-tau Yih
3. Gautier Izacard
4. Akari Asai
5. Patrick Lewis
6. Edouard Grave
7. Hannaneh Hajishirzi
8. Fabio Petroni

In Natural Language Processing

1. Sebastian Riedel
2. Hannaneh Hajishirzi
3. Luke S. Zettlemoyer
4. Wen-tau Yih
5. Edouard Grave
6. Douwe Kiela
7. Mohit Bansal
8. Patrick Lewis

Citations in the corpus

• Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Izacard et al. (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.”
• PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “A similar procedure can be used to filter the background corpus for a retrieve-and-read model (Izacard et al., 2020).”
• Efficient Passage Retrieval with Hashing for Open-domain Question Answering (2021, methodology): “Concurrent with our work, Izacard et al. (2020) attempted to reduce the memory cost of DPR using post-hoc product quantization with dimension reduction and filtering of passages.”
• Boosted Dense Retriever (2021, background): “Izacard et al. (2020) and Yang and Seo (2021) ex-periment with post-hoc compression and lower-dimensional embeddings, Lewis et al. (2021) index and retrieve question-answer pairs and Yamada et al. (2021) propose BPR, which approximates MIPS using binary vectors.”
• Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Izacard et al. (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.”
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “One of the earliest related works is the study by Izacard et al. (2020), who explore dimension reduction, product quantization, and passage filtering for single-vector retrievers.”
• Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021, unknown): “Bi-encoders solutions may be sub-optimal and memory ineffi-cient although memory-efficient dense retrieval has recently received attention (Izacard et al., 2020; Min et al., 2021; Lewis et al., 2021).”