Text and Code Embeddings by Contrastive Pre-Training

By Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy and 15 more. 2022

613 citations recorded by the source corpus.

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SGPT: GPT Sentence Embeddings for Semantic Search (2022, methodology): “We follow recent work [8, 11, 16] and perform supervised contrastive learning with in-batch negatives.”
Augmenting large language models with chemistry tools (2023, background): “The CAS number serves as a precise and universally recognized chemical identifier, enabling researchers to access relevant data and resources with ease, and ensuring that they obtain accurate and consistent information about the target molecule 89 .”
Improving Text Embeddings with Large Language Models (2023, methodology): “To address this challenge, methods like Contriever [17], OpenAI Embeddings [29], E5 [44], and BGE [46] adopt a multi-stage training paradigm.”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Typical methods to construct text pairs include Inverse Close Task (ICT) [9], random cropping [28] and neighboring text spans [41], etc.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Besides, it has shown that unsupervised contrastive pretraining [245] (pretraining with text pairs and text-code pairs) can also improve both text and code retrieval performance.”
Representation Learning with Large Language Models for Recommendation (2023, background): “The evaluation results reveal several interesting observations, as outlined below: • Overall, we consistently observe that integrating RLMRec with the backbone recommenders leads to improved performance compared to the original versions.”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “Internal memory is directly used for accomplishing zero-shot tasks [6, 7] while accessing external memory requires actions by the agent to fetch and integrate data into short-term memory for immediate use [149, 150].”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, methodology): “In recent years, continual progress has been achieved in this field, such as work from Contriever (Izacard et al., 2021), E5 (Wang et al., 2022b), and Ope-nAI Text Embedding (Neelakantan et al., 2022).”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “Alternatively, cpt-text (Neelakantan et al., 2022) passes pre-trained GPT decoders through a two-stage process using last token pooling to provide embeddings from decoders.”