Large Dual Encoders Are Generalizable Retrievers

By Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang and 1 more. 2021

644 citations recorded by the source corpus.

It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual encoder, where the final score is simply a dot-product between a query vector and a passage vector, is too limited compared to models with fine-grained interactions between the query and the passage. In this paper, we challenge this belief by scaling up the size of the dual encoder model while keeping the bottleneck layer as a single dot-product with a fixed size. With multi-stage training, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization. We further analyze the impact of the bottleneck layer and demonstrate diminishing improvement when scaling up the embedding size. Experimental results show that our dual encoders, Generalizable T5-based dense Retrievers (GTR), outperform previous sparse and dense retrievers on the BEIR dataset significantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10% of MS Marco supervised data to match the out-of-domain performance of using all supervised data.

Subfields: Artificial Intelligence, General Social Sciences, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Information Retrieval & Search

Citations in the corpus

Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, methodology): “A setup which is increasingly common in the neural retrieval community (Ni et al., 2021; Hofstätter et al., 2022).”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, neutral): “In general, these embedding models are first initialized from pre-trained BERT (Wang et al., 2022; Izacard et al., 2021) or T5 encoders (Ni et al., 2021).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “, 2023) begins by employing a passage retriever, such as Generalizable T5-based Retrievers (Ni et al., 2022), to gather pertinent information.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “For this, we use an existing query-document relevance model trained following Ni et al. (2021), which computes a relevance score Srelevance(q, e) between a query q and an evidence snippet e.”
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2021, methodology): “We show C ON QRR outperforms existing QR models on a recent large-scale open-domain CQA dataset QReCC (Anantha et al., 2021) by over 12% and 14% for BM25 and a neural dual encoder re-triever model (Ni et al., 2021) respectively, averaging over three retrieval metrics.”
Task-aware Retrieval with Instructions (2022, methodology): “…(Thakur et al., 2021) and LOTTE-pooled (Santhanam et al., 2022), outper-forming systems using three times more parameters (Nogueira et al. 2020; Ni et al. 2021; Muennighoff 2022) as well as task-specific retrievers trained on millions of automatically generated examples (Dai et al., 2022; Wang…”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “We use dense retrievers for Wikipedia, including GTR (Ni et al., 2022) and DPR (Karpukhin et al., 2020); we use BM25 for Sphere.”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “We implement P ROMPTAGATOR ’s dual encoders following GTR (Ni et al., 2021); in particular, we use a shared Transformer encoder initialized from T5, take the mean pooling of the top encoder layer, and project it to a ﬁxed 768-dimensional embedding.”
Multilingual E5 Text Embeddings: A Technical Report (2024, background): “Despite their significance, most existing embedding models are trained exclusively on English text (Reimers and Gurevych, 2019; Ni et al., 2022b,a), thereby limiting their applicability in multilingual contexts.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “training + Mutual distillation 2021 AR2-G [121] ERNIEbase SR IOC DyHN Adversarial training 2021 RepCONC [122] BERTbase SR IOC DyHN Quantization 2021 RocketQAv2 [38] ERNIEbase SR KL SHN+DeHN KD Joint training + Mutual distillation 2021 SPAR [123] BERTbase SR NLL SHN KD 2021 SEED [124] BERTbase SR NLL”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, methodology): “In recent years, there has been a continual effort in this field, where a series of well-known works are proposed, like Contriever (Izacard et al., 2021), GTR (Ni et al., 2021b), sentence-T5 (Ni et al., 2021a), Sentence-Transformer (Reimers and Gurevych, 2019), E5 (Wang et al., 2022a), Ope-nAI text ”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, methodology): “Unlike the original RARR implementation (Gao et al., 2023a), which uses the GTR retrieval model (Ni et al., 2022), we instead implement the scorer via few-shot LM prompting, as suggested by the authors.”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, background): “Several papers including (Ni et al., 2021a,b; Wang et al., 2022; Li et al., 2023) have shown that fine-tuning on these datasets leads to improvements in downstream performance.”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “…MARCO (Izacard et al., 2022a); for biographies and open-domain QA, we use additional top five documents retrieved by a web search engine, following Luo et al. (2023); for ASQA, we use the author-provided top 5 documents by GTR-XXL (Ni et al., 2022) across all baselines for a fair comparison.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “Furthermore, it is worth noting that our unsupervised pre-trained model narrows the gap even further with larger supervised base-lines, such as GTR and Sentence-T5.”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, methodology): “When compared to the GTR-XXL retriever, which also uses a model with billions of parameters based on the T5encoder (Ni et al., 2022), our model achieves higher MRR@10 and Recall@1k on the dev queries and on TREC DL19/DL20.”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “2021) on the MS MARCO dataset (Nguyen et al., 2016).”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, positive): “We consider two fine-tuned encoders, the aforementioned Contriever-ft, which contains 110M parameters, and the much larger GTR-XL model (Ni et al., 2022) with 1.”
C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “In recent years, there has been a continual effort in this field, where a series of well-known works are proposed, like Contriever [22], GTR [40], sentence-T5 [39], Sentence-Transformer [46], E5 [52], OpenAI text embedding [37], etc.”
REPLUG: Retrieval-Augmented Black-Box Language Models (2023, methodology): “Following prior work (Qu et al., 2021; Izacard & Grave, 2021b; Ni et al., 2021), we use a dense retriever based on the dual encoder architecture, where an encoder is used to encode both the input context x and the document d .”