Boosted Dense Retriever

By Patrick Lewis, Barlas Oğuz, Wenhan Xiong, Fabio Petroni, Wen-tau Yih, Sebastian Riedel. 2021

18 citations recorded by the source corpus.

We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble. The final representation is the concatenation of the output vectors of all the component models, making it a drop-in replacement for standard dense retrievers at test time. DrBoost enjoys several advantages compared to standard dense retrieval models. It produces representations which are 4x more compact, while delivering comparable retrieval results. It also performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. In practice, this can make the difference between serving indices from disk versus from memory, paving the way for much cheaper deployments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “Moreover, SEAL also performs better than MT-DPR (multi-task DPR) even when the latter is pretrained on tens of millions of questions from PAQ (Lewis et al., 2021c), a technique that can drastically improve results and that could potentially bring beneﬁts to our method as well (a task we leave for fu”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “training + Mutual distillation 2021 AR2-G [121] ERNIEbase SR IOC DyHN Adversarial training 2021 RepCONC [122] BERTbase SR IOC DyHN Quantization 2021 RocketQAv2 [38] ERNIEbase SR KL SHN+DeHN KD Joint training + Mutual distillation 2021 SPAR [123] BERTbase SR NLL SHN KD 2021 SEED [124] BERTbase SR NLL”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “Well-studied low hanging fruits for good tradeoffs include reducing the number of dimensions and reducing the number of bytes with quantization [12, 19, 25, 34].”