NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

By Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, B. Catanzaro, Bryan Catanzaro, Wei Ping, Wei Ping. International Conference on Learning Representations, 2024

542 citations recorded by the source corpus.

Decoder-only LLM-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce NV-Embed, incorporating architectural designs, training procedures, and curated datasets to significantly enhance the performance of LLM as a versatile embedding model, while maintaining its simplicity and reproducibility. For model architecture, we propose a latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling or using the lasttoken embedding from LLMs. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For training algorithm, we introduce a two-stage contrastive instruction-tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. For training data, we utilize the hard-negative mining, synthetic data generation and existing public available datasets to boost the performance of embedding model. By combining these techniques, our NV-Embed-v1 and NV-Embed-v2 models obtained the No.1 position on the MTEB leaderboard (as of May 24 and August 30, 2024, respectively) across 56 tasks, demonstrating the sustained effectiveness of the proposed methods over time. It also achieved the highest scores in the Long Doc section and the second-highest scores in the QA section of the AIR Benchmark, which covers a range of out-of-domain information retrieval topics beyond those in MTEB. We further provide the analysis of model compression techniques for generalist embedding models.

Subfields: Artificial Intelligence, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/cde48b24264e44355abb0e548a2cf7c70bb072b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Large Language Models for Information Retrieval: A Survey (2023, unknown): “This intriguing area remains open for further investigation and advancement.”
Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025, unknown): “Additionally, there has been considerable research facilitating the integration of LLMs into processes such as training data synthesis and quality data filtering (Wang et al., 2024; Lee et al., 2024; 2025b).”
Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “For teacher models, we compare NVEmbed v1 (Lee et al., 2024a), Arctic Embed Large (Merrick et al., 2024), and Stella 1.5B v5 (Zhang & FulongWang, 2024).”
Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (2026, unknown): “…recipe for adapting decoder-only LLMs into bi-encoders (Ma et al., 2024), and subsequent work explored LLM-generated training data (Wang et al., 2024), bidirectional attention (BehnamGhader et al., 2024), latent-attention pooling (Lee et al., 2025a), and multi-stage training (Zhang et al., 2025).”
A Systematic Multi-Domain Evaluation of Document Retrievers (2026, unknown): “Retrievers. docT5query [58] expands documents by generating multiple queries for each and retrieving with BM25. query2doc [80] instead expands the query, generating a pseudo-document and again retrieving with BM25, while HyDE [19] generates a hypothetical document and retrieves with the Contriever.”
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings (2026, unknown): “Recent embedding systems adapt decoder-only language models into bidirectional encoders and train them with large-scale contrastive objectives (BehnamGhader et al., 2024; Wang et al., 2024; Lee et al., 2025a; Kolodin et al., 2025).”
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs (2026, unknown): “The embedding models span two architectural sub-classes: encoder-based embedders built on bidirectional encoder backbones (e.g., E5 [72], BGE [8]) and decoder-based embedders built on causal LLM backbones (e.g., SFR-Embedding [45], Qwen3-Embedding [77], NV-Embed-v2 [35]).”
Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus (2026, unknown): “This technique elicits a sentence-level sum-mary representation at the position immediately preceding the prospective one-word answer; • NV-Embed 2 ( nvidia/NV-Embed-v2 ), developed by Nvidia [33], based on a latent attention layer, decoder-only 7.85-”
HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation (2026, unknown): “More recently, decoder-only embedding models initialize from LLMs to exploit their pretrained linguistic knowledge (Muennighoff et al., 2024; Lee et al., 2024; Zhang et al., 2025).”
EmbeddingRWKV: State-Centric Retrieval with Reusable States (2026, unknown): “…from early BERT-based encoders (Devlin et al., 2019; Hou et al., 2024a) to large language model backbones, with recent systems such as NV-Emb (Lee et al., 2024), E5-Mistral (Wang et al., 2024c), and GTE (Zhang et al., 2024) achieving strong performance on benchmarks like MTEB (Muennighoff et…”
Efficient Temporal-aware Matryoshka Adaptation for Temporal Information Retrieval (2026, unknown): “The prompt includes few-shot examples drawn from the original TNP dataset, with temporal expressions extracted using SUTime.”
Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (2025, unknown): “5-7B (Li et al., 2023); (2) Commercial Late-interaction: Voyage-Context-3 (Voyage-AI, 2025); (3) Latent-Attention Em-bedding: NV-Embed-v2-7B (Lee et al., 2025), a general-purpose embedding model employing latent attention layers; (4) Context-Aware SOTA: SitEmb-8B (Wu et al., 2025), which encodes chu”
GigaEmbeddings: Efficient Russian Language Embedding Model (2025, unknown): “(Lee et al., 2025) This hierarchical approach, from large-scale pre-training to task-specialized fine-tuning and multitask generalization, ensures that the model first learns broad semantic patterns, then hones discriminative precision and finally achieves cross-task robustness.”
ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “As backbone generative models improved, retriever performance improved as well showcasing the central impact of language model pretraining and modality alignment (Xu et al., 2025; Nussbaum et al., 2025).”
Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “There has been a recent explosion of LLM-based embedding models such as SFR-Embedding-Mistral (Meng et al., 2024), NV-Embed (Lee et al., 2024), bge-en-icl (Li et al., 2024), gte-Qwen2-1.”
SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension (2025, unknown): “2) LLM-based embedding models : E5-Mistral ( E5-Mistral-7b-Instruct ) (Wang et al., 2024a), GTE-Qwen2 ( GTE-Qwen2-7b-Instruct ) (Li et al., 2023), NV-Embed-v2 (Lee et al., 2024) and Qwen3-Embedding-8B (Zhang et al., 2025).”
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “…embedding models, which produce task-specific embeddings by incorporating natural language instructions or task descriptions, have demonstrated remarkable performance on various tasks (Su et al., 2023; Asai et al., 2023; Lee et al., 2024a,b; Wang et al., 2022; Li et al., 2023; Xiao et al., 2024).”