Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

By Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Matthew Cer, Yinfei Yang. Findings, 2021

830 citations recorded by the source corpus.

We provide the first exploration of sentence embeddings from text-to-text transformers (T5) including the effects of scaling up sentence encoders to 11B parameters. Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods to construct Sentence-T5 (ST5) models: two utilize only the T5 encoder and one using the full T5 encoder-decoder. We establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our encoder-only models outperform the previous best models on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Scaling up ST5 from millions to billions of parameters shown to consistently improve performance. Finally, our encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/dbe87b171bfb789e1d22a047aeeee69105e6fd02

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Transformer Memory as a Differentiable Search Index (2022, methodology): “Dual encoders (Dehghani et al., 2017; Gillick et al., 2018; Gao et al., 2021; Ni et al., 2021; Karpukhin et al., 2020) are a well-established paradigm for retrieval.”
Recommender Systems with Generative Retrieval (2023, methodology): “For example, if two items share the Semantic ID ( 12 , 24 , 52 ) , we append extra tokens to differentiate between them, hence they are represented as ( 12 , 24 , 52 , 0 ) and ( 12 , 24 , 52 , 1 ) .”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Although these datasets are small, existing works [43, 44] have shown that supervised fine-tuning leads to consistent performance gains.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “index 2021 Condenser [112] Condenser SR CE IB REP 2021 DRPQ [113] BERTbase MR NLL IB+DyHN Multiple pseudo query embeddings 2021 RANCE [114] RoBERTabase SR NLL DeHN 2021 DANCE [115] RoBERTabase SR IOC SHN Contrastive dual learning 2021 DPR-PAQ [116] RoBERTalarge SR NLL IB+SHN ALD GAP 2021 JPQ [117] B”
Large Dual Encoders Are Generalizable Retrievers (2021, methodology): “Following Ni et al. (2021), we encode q i and p + i into embeddings by feeding them to the T5 encoder and taking the mean pooling of the encoder output.”
C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “In recent years, there has been a continual effort in this field, where a series of well-known works are proposed, like Contriever [22], GTR [40], sentence-T5 [39], Sentence-Transformer [46], E5 [52], OpenAI text embedding [37], etc.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “…leveraging large quantities of unlabeled web data through unsupervised contrastive pre-training, coupled with task-specific data, prompts, or instructions to mitigate task conflicts during fine-tuning (Ni et al., 2022a,b; Neelakantan et al., 2022; Wang et al., 2022b; Su et al., 2023).”
Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders (2024, neutral): “We evaluate 11 top-performing LLMs from the widely used general text embedding benchmark MTEB (Muennighoff et al., 2023; Enevoldsen et al., 2025): (1) open-source LLMs with < 1 B parameters (RoBERTa (Liu et al., 2019), SimCSE (Gao et al., 2021), Sentence-T5 (Ni et al., 2022)); (2) open-source LLMs w”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, positive): “One major driving force is the popularity of pre-trained language models, where the underlying semantic of the data can be effectively encoded by such powerful text encoders (Reimers and Gurevych, 2019; Karpukhin et al., 2020; Ni et al., 2022).”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, methodology): “T5 (Text-to-Text Transfer Transformer) [84] is a ground-breaking LLM developed by Google Research, revolutionizing NLP tasks.”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “We employ two standard models for retrieval: first, we use a dual encoder (Reimers & Gurevych, 2019; Karpukhin et al., 2020; Ni et al., 2021) to select an initial set of candidates.”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “GTR (Ni et al., 2021b) and ST5 (Ni et al., 2021a) are based on the encoder part of the T5 model (Raffel et al., 2020) and only differ in their fine-tuning datasets.”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “The second class of baselines focuses on semantic textual similarity: SimCSE (Gao et al., 2021), Sent-T5 (Ni et al., 2022), and SGPT-NLI (Muennighoff, 2022).”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, result): “Similar to Wang et al. (2022); Li et al. (2023); Xiao et al. (2023); Ni et al. (2022), we use large collections of publicly available data to form pairs.”
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2021, methodology): “, 2021) by over 12% and 14% for BM25 and a neural dual encoder retriever model (Ni et al., 2021) trained on the standard MSMARCO retrieval dataset (Nguyen et al.”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, methodology): “In recent years, there has been a continual effort in this field, where a series of well-known works are proposed, like Contriever (Izacard et al., 2021), GTR (Ni et al., 2021b), sentence-T5 (Ni et al., 2021a), Sentence-Transformer (Reimers and Gurevych, 2019), E5 (Wang et al., 2022a), Ope-nAI text ”