Unsupervised Dense Information Retrieval with Contrastive Learning
By Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave. 2021
1,593 citations recorded by the source corpus.
Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS~MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Sebastian Riedel
- Graham Neubig
- Wen-tau Yih
In Computer Vision and Pattern Recognition
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Luc Van Gool
- Mohit Bansal
- Dacheng Tao
- Jianfeng Gao
- Trevor Darrell
- Sebastian Riedel
In Information Retrieval & Search
- Wen-tau Yih
- Sebastian Riedel
- Jimmy J. Lin
- Hannaneh Hajishirzi
- Luyu Gao
- Jamie Callan
- Danqi Chen
- Gautier Izacard
In Representation & Self-Supervised Learning
- Percy Liang
- Luke S. Zettlemoyer
- Sebastian Riedel
- Noah A. Smith
- Akari Asai
- Niklas Muennighoff
- Timo Schick
- Hannaneh Hajishirzi
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Atlas retrieves relevant documents based on the current context by using a general-purpose dense retriever using a dual-encoder architecture, based on the Contriever (Izacard et al., 2022).”
- Lost in the Middle: How Language Models Use Long Contexts (2023, methodology): “When retrieving from Wikipedia to answer queries from NaturalQuestions-Open, we find that model performance saturates long before retriever re-call saturates, indicating that current models fail to effectively use additional retrieved doc-uments—using 50 documents instead of 20 retrieved documents o”
- Large Language Models for Information Retrieval: A Survey (2023, neutral): “Then, a TART-dull model initialized from Contriever [134] is learned by distillating knowledge from the TART-full.”
- Representation Learning with Large Language Models for Recommendation (2023, methodology): “RLMRec utilizes contrastive and generative alignment techniques to align CF-side relational embeddings with LLM-side semantic representations, effectively reducing feature noise.”
- Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2021, methodology): “Regardless of the choice of the base dense retriever, adding either the BM25 or UniCOIL Λ can consistently and substantially boost the performance of the base retriever, even for very recent SoTA models such as Contriever (Izacard et al., 2022) and GTR (Ni et al., 2021).”
- C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “While one can resort to generic pre-trained models like BERT [15] and T5 [45], the quality of text embedding can be substantially improved by pre-training with large-scale unlabeled data [22, 53].”
- Task-aware Retrieval with Instructions (2022, methodology): “The first line of work uses purely unsupervised approaches, such as pre-training neural retrievers on synthetic data (e.g., Contriever; Izacard et al. 2022).”
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “By default, we use the top five documents from Contriever-MS MARCO (Izacard et al., 2022a); for biographies and open-domain QA, we use additional top five documents retrieved by a web search engine, following Luo et al. (2023); for ASQA, we use the author-provided top 5 documents by GTR-XXL (Ni et a”
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, methodology): “…augmenting LMs with non-parametric memories (i.e., retrieved text chunks) enables much smaller models to match the performance of larger models (Izacard et al., 2022b; Khandelwal et al., 2020; Min et al., 2022), although Chen et al. (2022) and Longpre et al. (2021) show that even those models…”
- Cognitive Architectures for Language Agents (2023, background): “, 2023a) or retrieval distillion (Izacard et al., 2021) may be helpful for language agents to learn better retrieval procedures.”
- LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “Each example in the dataset contains a question and k related documents, utilizing the Contriever retrieval system (Izacard et al., 2022), one of which includes a document with the correct answer.”
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “…and SimCSE (Gao et al., 2021) are applied solely to text similarity and classification tasks, while DPR (Karpukhin et al., 2020) and Contriever (Izacard et al., 2022) Instruction Finetuning Recent work demonstrated that instruction-finetuned language models could perform new tasks given a…”
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, neutral): “In Table 9, we compare the performance of Stella V5 with two alternative popular retrievers: BM25 (Robertson & Zaragoza, 2009) and Contriever (Izacard et al., 2022).”
- ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “The retrieval model can be trained standalone [44] [45] [46] or jointly with language models [47] [42].”
- The Faiss Library (2024, neutral): “For this paper, we introduce a dataset of 768-dimensional Contriever text embed-dings [4] that are compared using inner product similarity.”
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “Then, they are further pre-trained with contrastive learning on curated unsupervised (Izacard et al., 2021) or weakly-supervised text pairs (Wang et al., 2022).”