SGPT: GPT Sentence Embeddings for Semantic Search

By Niklas Muennighoff. 2022

265 citations recorded by the source corpus.

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Contrastive Finetuning We also perform contrastive finetuning of the 1.3 and 7.1 billion parameter BLOOM models using the SGPT Bi-Encoder recipe (Muennighoff, 2022) to train models that produce high-quality text embeddings.”
C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “According to previous studies, the learning of the embedding model may benefit from the increasing of negative samples [22, 32, 43].”
Task-aware Retrieval with Instructions (2022, methodology): “…al., 2021) and LOTTE-pooled (Santhanam et al., 2022), outper-forming systems using three times more parameters (Nogueira et al. 2020; Ni et al. 2021; Muennighoff 2022) as well as task-specific retrievers trained on millions of automatically generated examples (Dai et al., 2022; Wang et al., 2022a).”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “According to previous studies, the learning of the embedding model may benefit from the increasing of negative samples (Izacard et al., 2021; Qu et al., 2020; Muennighoff, 2022).”
Scaling Data-Constrained Language Models (2023, background): “These models are generally pre-trained to predict the next token in a sequence, which makes them applicable to various language tasks directly after pre-training [15, 115, 50, 69, 99] by reformulating said NLP tasks as context continuation tasks (see [65] for an earlier proposal on this topic).”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “When applying (weighted) mean pooling (Muennighoff, 2022), we exclude the instruction tokens.”
Large Language Models for Information Retrieval: A Survey (2023, unknown): “This intriguing area remains open for further investigation and advancement.”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “5 All of these baselines are based on pretrained language models, achieving strong performance on the MTEB leader-board.”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “(b) Transformer decoder methods SGPT BiEncoders (Muennighoff, 2022) perform contrastive fine-tuning of <0.”
Improving Text Embeddings with Large Language Models (2023, background): “SGPT [26], GTR [31], and Udever [49] demonstrate the scaling law of text embeddings empirically, but their performance still falls behind small bidirectional encoders such as E5 [44] and BGE [46].”
What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models (2025, unknown): “Beyond re-ranking, mLLMs have been used to reason directly over candidate passages, or to generate synthetic training data for retrievers [50, 51, 52]. mLLM-based re-ranking methods can be categorised as listwise, pairwise or pointwise.”
Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework (2026, unknown): “Historically decoders were seen as less natural for embeddings (since they’re built for next-word generation, not bidirectional understanding), so this family tests whether large generative models can be adapted into competitive embedding models simply by pooling their internal representations [20].”
Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models (2025, unknown): “…like TF-IDF and BM25 (Robertson et al., 2009) were traditionally used for retrieval, dense retrievers like transformer architectures (Vaswani et al., 2017) are now widely used for semantic search (Cer et al., 2018; Yates et al., 2021; Thakur et al., 2021; Muennighoff, 2022; Nussbaum et al., 2024).”
Mieb: Massive Image Embedding Benchmark (2025, unknown): “E5-V uses pre-trained MLLMs followed by text-only contrastive fine-tuning with prompts like “sum-marize the above sentence with one word” and last-token pooling [40, 44], showing surprising generalization to images and interleaved encodings.”