LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

By Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, Siva Reddy. arXiv.org, 2024

526 citations recorded by the source corpus.

Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 4 popular LLMs ranging from 1.3B to 8B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data (as of May 24, 2024). Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/18594d3f15b39385bace39657d14f0c8c7479db1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hayato Tsukagoshi
2. Hong-Liu Cao
3. Ryohei Sasano

Citations in the corpus

• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, unknown): “It is important to note that certain datasets (e.g., MSMARCO) are training splits of the MTEB Benchmark, which we follow the existing practices established by leading generalist embedding models (Meng et al., 2024b; Wang et al., 2023b; BehnamGhader et al., 2024; Muennighoff et al., 2024).”
• Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (2026, unknown): “…recipe for adapting decoder-only LLMs into bi-encoders (Ma et al., 2024), and subsequent work explored LLM-generated training data (Wang et al., 2024), bidirectional attention (BehnamGhader et al., 2024), latent-attention pooling (Lee et al., 2025a), and multi-stage training (Zhang et al., 2025).”
• Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings (2026, unknown): “Recent embedding systems adapt decoder-only language models into bidirectional encoders and train them with large-scale contrastive objectives (BehnamGhader et al., 2024; Wang et al., 2024; Lee et al., 2025a; Kolodin et al., 2025).”
• LLMs Meet Isolation Kernel: Lightweight, Learning-free Binary Embeddings for Fast Retrieval (2026, unknown): “More recently, there has been a paradigm shift toward leveraging large language models (LLMs) for text embeddings, with notable examples including LLM2Vec (BehnamGhader et al., 2024), Llama2Vec (Liu et al., 2024b), and Qwen Embedding (Li et al., 2023; Zhang et al., 2025b).”
• Minimizing Human Intervention in Online Classification (2025, unknown): “Recently, the NLP community has increasingly favored decoder architectures for creating embeddings (Springer et al., 2024; BehnamGhader et al., 2024), as these approaches have demonstrated superior performance over traditional encoder-based methods.”
• GigaEmbeddings: Efficient Russian Language Embedding Model (2025, unknown): “Similarly, LLM2Vec (BehnamGhader et al., 2024) transforms decoder-only models such as LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023) into universal text encoders via parameter-efficient meth-ods (Xu et al., 2023), demonstrating that causal LLMs can rival traditional encoder-only archi”
• ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “Through controlled experiments—ranging from language model pretraining to multi-stage, domain-specific fine-tuning, we aim to answer a central question: Which design choices best boost performance in modern visual document retrievers?”
• Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “…of this gap is due to the sentiment within the community that decoders can be adapted for use in tasks that were once predominantly encoder-focused (e.g. classification, embeddings), especially as they can often be used in a zero-shot fashion (i.e. without fine-tuning) (BehnamGhader et al., 2024).”
• Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “Notably, one common modification is the use of bidirectional attention (BehnamGhader et al., 2024; Lee et al., 2024a; de Souza P. Moreira et al., 2024).”
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Recent advancements in large language models (LLMs) have demonstrated their effectiveness and robustness in text retrieval tasks (Muennighoff et al., 2024; Sun et al., 2023; Li et al., 2025; BehnamGhader et al., 2024; Lee et al., 2025).”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “LLM2V EC (BehnamGhader et al., 2024) enables bidirection and further trains L LAMA -2 (Touvron et al., 2023) with specific adaptive tasks.”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “By addressing common retrieval failures using synthetic data and leveraging instruction tuning [145, 172], the model achieves state-of-the-art results on the MTEB benchmark [109], outperforming proprietary embeddings from OpenAI and Cohere on English tasks and surpassing…”
• Ruri: Japanese General Text Embeddings (2024, methodology): “To address this, in addition to collecting and preprocessing existing datasets, we synthesized additional training data using Large Language Models (LLMs) for contrastive pre-training.”
• Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, methodology): “…sampling: in-batch negative samples and mined hard negatives • Model size: 7B d) LLM2Vec : Similar to the idea of Echo-mistral, the authors of [39] believe that the slow adoption of decoder-only LLMs in text embedding tasks is partly due to their causal attention mechanism, which restricts…”
• Bagging-Based Model Merging for Robust General Text Embeddings (2026, unknown): “To address this, LLM2vec[4] replaced causal attention with bidirectional attention and introduced a masked next-token prediction (MNTP) warm-up strategy.”
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval (2024, unknown): “Query Question A summary of the code could be as following: A method that returns the average color of pixels in a neighborhood around a given pixel in an image.”
• CoRet: Improved Retriever for Code Editing (2025, unknown): “BehnamGhader et al. (2024) proposes LLM2Vec which allows LLMs to be used for dense retrieval.”