BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
By Nandan Thakur, Nils Reimers, Andreas Ruckl'e, Andreas Ruckl'e, Abhishek Srivastava, Iryna Gurevych, I. Gurevych. 2021
1,831 citations recorded by the source corpus.
Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Hannaneh Hajishirzi
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Yejin Choi
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Mohit Bansal
- Andrew Zisserman
- Xiaogang Wang
- Jitendra Malik
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Wen-tau Yih
- Sebastian Riedel
- Luyu Gao
- Omar Khattab
- Sebastian Hofstätter
- Gautier Izacard
In Evaluation & Benchmarks
- Hannaneh Hajishirzi
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Sebastian Riedel
- Kyunghyun Cho
- Jimmy J. Lin
- Danqi Chen
Citations in the corpus
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Unfortunately, in this setting, dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al., 2021).”
- Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “Few-shot and zero-shot retrieval will also be of increasing importance, and there are already works investigating this direction (Maillard et al., 2021; Thakur et al., 2021).”
- Improving Passage Retrieval with Zero-Shot Question Generation (2022, background): “Table 10: UPR re-ranking results on the BEIR benchmark (Thakur et al., 2021).”
- RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “We note several exciting future directions for R ANK G EN , including: • training (or adapting) a multilingual variant of R ANK G EN , as our current models are trained on English text only • further exploring the capability of R ANK G EN as a retriever, either zero-shot or by fine-tuning on retrieva”
- Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “BEIR Benchmark [53] is a collection of 19 information retrieval datasets, ranging across ad-hoc web search, question answering, fact verification and duplicate question retrieval, etc.”
- NaturalProofs: Mathematical Theorem Proving in Natural Language (2021, background): “This aligns with recent findings about neural retrieval models in various zero-shot settings [39].”
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, background): “The observations are further validated on seven BEIR datasets covering various domains, where PRP performs competitively with supervised rankers and outperforms other LLM based approaches by a large margin.”
- Tracing Knowledge in Language Models Back to the Training Data (2022, methodology): “Among these approaches, BM25 (Robertson et al., 1995; Lv and Zhai, 2011), the best performing variant, has been consistently used as a strong baseline for information retrieval benchmarks (Thakur et al., 2021).”
- Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders (2024, neutral): “An early example is BEIR (Thakur et al., 2021), which evaluates text embedding models on information retrieval tasks.”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “The conducted experiments on BEIR show that the zero-shot retrieval capacity of dense retrieval models is highly limited compared to classic retrieval models such as BM25 [72].”
- MTEB: Massive Text Embedding Benchmark (2022, methodology): “MTEB reuses datasets and evaluation from BEIR (Thakur et al., 2021).”
- Text and Code Embeddings by Contrastive Pre-Training (2022, methodology): “We also fine-tune the unsupervised model on MSMARCO and evaluate it on a suite of zero-shot search tasks in the BEIR benchmark (Thakur et al., 2021).”
- Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (2025, positive): “The following benchmarks are particularly relevant: • BEIR (Benchmarking Information Retrieval): A benchmark for evaluating embedding models across 17 datasets in bioinformatics, finance, and question answering [58].”
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, background): “Text Embeddings Text embeddings are useful in many applications such as information retrieval (Thakur et al., 2021), text similarity (Gao et al., 2021), prompt retrieval for in-context learning (Su et al., 2022), classification (Reimers and Gurevych, 2019), and beyond.”
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “SFR-Embedding-2R (Meng et al., 2024b) demonstrates competitive scores on the MTEB (70.31) and BEIR (60.18) benchmarks by continuing to finetune the e5-mistral-7b-instruct model (Wang et al., 2023b).”
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, background): “However, several more sophisticated retrieval strategies exist, which are known to boost performance (Thakur et al., 2021) and could be useful here.”
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, positive): “We report retrieval results on the popular BEIR evaluation suite (Thakur et al., 2021), the common standard for evaluating retrieval performance across a variety of tasks and domains, using the nDCG@10 metric.”