Towards General Text Embeddings with Multi-stage Contrastive Learning

By Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang. arXiv.org, 2023

940 citations recorded by the source corpus.

We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/84109e1235b725f4bb44a54bab8b493bd723fdd3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, background): “…that consists of large-scale weakly supervised contrastive training followed by multi-task fine-tuning (Ni et al., 2022; Wang et al., 2022a; Li et al., 2023a; Xiao et al., 2023) Recent approaches have focused on enhancing the generalization and transferability of text embeddings using…”
Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025, positive): “The multi-stage training approach is a common practice for training text embedding models (Li et al., 2023; Wang et al., 2022; Chen et al., 2024).”
C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “Although there were many strong competitors in the English community, such as E5 [53], SGPT [32], GTE [28], GTR [40], and OpenAI Ada-002 [37], we were able to notably advance the prior SOTA by an absolute 1.1 points in total average, which further verify the effectiveness of our data curation and…”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “To address this consideration, we employ instructions in the GSM8k and MATH test set as queries to retrieve the top-5 samples from all evolved training data with an embedding model, gte-large (Li et al., 2023d).”
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “We construct them via contrastive learning (Wang et al., 2022; Li et al., 2023) scale meticulously curated datasets, providing robust off-the-shelf retrieval models.”
Temporal Preference Optimization for Unsupervised Retrieval (2026, unknown): “GTE (Li et al., 2023) improves generalization by training on diverse datasets, while M3-Embedding (Chen et al., 2024) uses self-distillation to unify signals from multiple retrieval paradigms.”
IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions (2026, unknown): “: InstructOR (Su et al., 2023), GritML-7B (Muennighoff et al., 2025), E5-Mistral (Wang et al., 2024b), the GTE models (Li et al., 2023), which are based on Qwen-2 (Yang et al., 2024), the SFR-Embedding-Mistral (Meng et al., 2024), and the Linq-Embed-Mistral (Choi et al., 2024), Lychee-embed (Zhang…”
Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry (2025, unknown): “…[18] Encoder-only Transformer (GTE English-specialized) [CLS] token or projected representation nomic-embed-text-v2-moe(0.5B) gte-Qwen2-7B-instruct [18] Decoder-only Transformer (Qwen2-7B backbone) Final [EOS] embedding e5-mistral-7b-instruct [25] Decoder-only Transformer (Mistral-7B, 32 layers,…”
On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey (2025, unknown): “O n T h e R o l e o f P L M s i n G P T E Background (§2)”
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation (2025, unknown): “Retrieval-oriented embedding families include E5 [56], GTE [57], and BGE [58].”
A Systematic Multi-Domain Evaluation of Document Retrievers (2026, unknown): “Retrievers. docT5query [58] expands documents by generating multiple queries for each and retrieving with BM25. query2doc [80] instead expands the query, generating a pseudo-document and again retrieving with BM25, while HyDE [19] generates a hypothetical document and retrieves with the Contriever.”
Characterizing LLM scientific concept generation: A multi-dimensional measurement study (2026, unknown): “The evolution from models like Sentence-BERT [19] and all-mpnet-base-v2 to modern architectures including BGE [21], Nomic Embed [22], and GTE [23] has improved representation quality but also introduced model-dependent variation in distance distributions.”
REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling (2026, unknown): “5B-v5, Zhang et al., 2024), the GN backbones (GTE, Li et al., 2023b; BGE, Xiao et al., 2024) used directly under truncated and chunked mean-pool protocols, and a multi-GN REIGN sweep.”
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings (2026, unknown): “E5 (Wang et al., 2022), GTE (Li et al., 2023), BGE (Xiao et al., 2023), and Nomic Embed (Nussbaum et al., 2024) scaled data and negative mining.”