Improving Text Embeddings with Large Language Models

By Liang Wang, Nan Yang, Nan Yang, Xiaolong Huang, Lin-Jun Yang, Linjun Yang, Rangan Majumder, Fu-Ru Wei, Furu Wei. Annual Meeting of the Association for Computational Linguistics, 2023

400 citations recorded by the source corpus.

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/88ca4ee548d07263386ca8e4effc4a001bb2716f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Ping
2. Bryan Catanzaro
3. Hong-Liu Cao
4. Mohammad Shoeybi
5. Liang Wang
6. Chankyu Lee
7. Nan Yang
8. Fu-Ru Wei

Citations in the corpus

• Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, neutral): “The top two performing open-source long context embedding models are jina-embedding-v2-50 Nomic Embed Jina Base V2 text-embedding-3-small text-embedding-ada Figure 1: Text Embedding Model Benchmarks.”
• Large Language Models for Information Retrieval: A Survey (2023, neutral): “Unfortunately, there are some long-term challenges that hinder the advancement of retrieval models.”
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “Following the ablation studies in Moreira et al. (2024), we use E5-mistral-7b-instruct (Wang et al., 2023b) as a teacher retrieval model to identify the optimal hardnegative passages relevant to the query.”
• Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “Recent work has focused on scaling and data curation (Xiao et al., 2023; Wang et al., 2022; Li et al., 2023; G¨unther et al., 2023; Nussbaum et al., 2024; Merrick et al., 2024; Yu et al., 2024) or adapting decoder-only LLMs for embedding tasks (Wang et al., 2023; Lee et al., 2024b).”
• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “For open-source text embedding models, we include E5-Base (Wang et al., 2022) and E5-Mistral (Wang et al., 2023a), the two most performant text embedding models from the CoIR benchmark, as well as Arctic-Embed-M (Merrick et al., 2024), the base text encoder that we finetune on C O RNS TACK .”
• Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry (2025, unknown): “…token or projected representation nomic-embed-text-v2-moe(0.5B) gte-Qwen2-7B-instruct [18] Decoder-only Transformer (Qwen2-7B backbone) Final [EOS] embedding e5-mistral-7b-instruct [25] Decoder-only Transformer (Mistral-7B, 32 layers, 4096 dim) Hidden state of final token after instruction prefix.”
• Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings (2026, unknown): “Recent embedding systems adapt decoder-only language models into bidirectional encoders and train them with large-scale contrastive objectives (BehnamGhader et al., 2024; Wang et al., 2024; Lee et al., 2025a; Kolodin et al., 2025).”
• Granite Embedding Multilingual R2 Models (2026, unknown): “…2024; Zhang et al., 2024; Wang et al., 2024a; Nussbaum et al., 2024; Awasthy et al., 2025b) are widely used for retrieval due to their low inference latency and small memory footprint compared to decoder-based models (Lee et al., 2024; Wang et al., 2023; Zhang et al., 2025; Akram et al., 2026).”
• Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study (2026, unknown): “Recent LLM encoders (E5-Mistral-7B [22]) and commercial application programming interfaces (APIs; OpenAI text-embedding-3-small [23], hereafter OpenAI-emb3-small) have further expanded the landscape.”
• Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus (2026, unknown): “…hidden state at the EOS-token position, computed with attention-mask-aware last-token pooling so that padding does not affect the extracted vector [69]; • LLaMA 2 7B ( NousResearch/Llama-2-7b-hf ), developed by Meta [61, 62], based on a 32-layer 6.7B-parameter custom large language model, 4096…”
• Minimizing Human Intervention in Online Classification (2025, unknown): “Among these, Mistral-E5 (Wang et al., 2024) stands out as the state-of-the-art LLM for text retrieval.”
• Granite Embedding R2 Models (2025, unknown): “Similar to Wang et al. (2024), for each query-document pair ( q, p ) , we add an instruction template to the original query to generate a new one: where { task definition } is a placeholder for a one-sentence description of the embedding task.”
• ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings (2025, unknown): “This synthetic data approach has also been successfully applied to train text-embedding models, such as E5-Mistral-7B-instruct , which demonstrated that a general-purpose model could be trained almost exclusively on synthetic query-passage pairs [35].”
• SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension (2025, unknown): “…in embedding models, either by designing efficient bidirectional models (Chen et al., 2024; Sturua et al., 2024; Nussbaum et al., 2024), or by repurposing powerful unidirectional pre-trained LLMs as embedding generators (Li et al., 2023; Wang et al., 2024a; Moreira et al., 2024; Kim et al., 2024).”