Text Embeddings by Weakly-Supervised Contrastive Pre-training

By Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Lin-Jun Yang, Linjun Yang, Daxin Jiang, Daxin Jiang, Rangan Majumder, Furu Wei. arXiv.org, 2022

1,460 citations recorded by the source corpus.

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5a3c1afe73d8bcc8288d17cb17be2baec8a98464

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “2.1 B IDIRECTIONAL E MBEDDING M ODELS BERT (Devlin et al., 2018) or T5 (Raffel et al., 2020)-based embedding models have long been the dominant approaches for general-purpose embedding tasks.”
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “We construct them via contrastive learning (Wang et al., 2022; Li et al., 2023) scale meticulously curated datasets, providing robust off-the-shelf retrieval models.”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, methodology): “The majority of the top open-source models on the MTEB benchmark (Muennighoff et al., 2023) are limited to context lengths of 512, such as E5 Wang et al. (2022), GTE Li et al. (2023), and BGE Xiao et al. (2023).”
Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “The results are presented in Figure 4.”
MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024, neutral): “5 (Xiao et al., 2023), jina-embeddings-v2-base-en (Günther et al., 2023), e5-base-v2 (Wang et al., 2022), and instructor-large (Su et al., 2023).”
ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “First, we compare our domain adapted retrieval model with Sentence Transformer [34] and e5 small unsupervised [30] on each category.”
The MiniPile Challenge for Data-Efficient Language Models (2023, methodology): “First, we infer embeddings for all documents using E5-Large (EmbEddings from bidirEctional Encoder rEpresentations) [56], a state-of-the-art text embedding model, which achieves excellent performance on the MTEB benchmark [42].”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, positive): “For retrieval, we use the 2018 Wikipedia dump (Karpukhin et al., 2020) as the knowledge source and E5 (Wang et al., 2022) as the retriever.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “Surprisingly, these pipelines currently rely on older models, and quite often on the original BERT itself as their backbone (Wang et al., 2022; Xiao et al., 2023), without leveraging improvements developed in recent years.”
Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025, neutral): “The multi-stage training approach is a common practice for training text embedding models (Li et al., 2023; Wang et al., 2022; Chen et al., 2024).”
Improving Text Embeddings with Large Language Models (2023, methodology): “To address this challenge, methods like Contriever [17], OpenAI Embeddings [29], E5 [44], and BGE [46] adopt a multi-stage training paradigm.”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, background): “…learning pipeline that consists of large-scale weakly supervised contrastive training followed by multi-task fine-tuning (Ni et al., 2022; Wang et al., 2022a; Li et al., 2023a; Xiao et al., 2023) Recent approaches have focused on enhancing the generalization and transferability of text…”
C-Pack: Packed Resources For General Chinese Embeddings (2023, methodology): “While one can resort to generic pre-trained models like BERT [15] and T5 [45], the quality of text embedding can be substantially improved by pre-training with large-scale unlabeled data [22, 53].”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, positive): “One common form of embedding-based IR application is dense retrieval, where relevant answers to the query can be retrieved based on the embedding similarity (Karpukhin et al., 2020; Xiong et al., 2020; Nee-lakantan et al., 2022; Wang et al., 2022; Xiao et al., 2023).”
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (2023, methodology): “We explore two methods for encoding sentences: one involves using the representation from the model which will be trained after data selection, while the other employs well-trained sentence embedding models such as E5-Large-V2 (Wang et al., 2022).”
Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, methodology): “They further pre-train the models in a supervised [29] or contrastive manner [31, 32].”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “For further details (Wang et al., 2022b).”