Universal Sentence Encoder for English

By Daniel Matthew Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar and 2 more. 2018

1,019 citations recorded by the source corpus.

We present easy-to-use TensorFlow Hub sentence embedding models having good task transfer performance. Model variants allow for trade-offs between accuracy and compute resources. We report the relationship between model complexity, resources, and transfer performance. Comparisons are made with baselines without transfer learning and to baselines that incorporate word-level transfer. Transfer learning using sentence-level embeddings is shown to outperform models without transfer learning and often those that use only word-level transfer. We show good transfer task performance with minimal training data and obtain encouraging results on word embedding association tests (WEAT) of model bias.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, background): “These embeddings may be at the word (Mikolov et al., 2013), sentence (Cer et al., 2019), or paragraph level (Le & Mikolov, 2014).”
Generating Datasets with Pretrained Language Models (2021, methodology): “…S-RoBERTa (base) trained on datasets generated with D INO to S-BERT and S-RoBERTa ﬁnetuned on NLI data as well as Universal Sentence Encoder (USE) (Cer et al., 2018) Table 2: Effect of removing self-debiasing ( λ = 0 ) or increasing the decay constant ( λ = 200 ), using no label smoothing and…”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “…bank of web documents and construct sentence embeddings (Kiros et al., 2015; Wieting et al., 2016; Conneau et al., 2017; Artetxe and Schwenk, 2019; Cer et al., 2018; Arora et al., 2017) that allow us to retrieve domain-speciﬁc unannotated sentences, which are similar to the existing training set…”
Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “For our next analysis, we use sentence embeddings from the Universal Sentence Encoder [18] to compare how the semantic content of captions differs between lighter and darker images.”
On Measuring Social Biases in Sentence Encoders (2019, methodology): “The model is trained on SNLI, Wikipedia, web news, and other online sources (Cer et al., 2018).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, positive): “This was later extended to multitask learning, combining different training objectives like that of skip-thought, NLI, and machine translation (Cer et al., 2018; Subramanian et al., 2018).”
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (2019, methodology): “MLP: A multi-layer perceptron with USE embeddings (Cer et al., 2018) as input.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “…that can be used without finetuning for downstream tasks (Le & Mikolov, 2014; Kiros et al., 2015; Wieting et al., 2016; Hill et al., 2016; Arora et al., 2017; Conneau et al., 2017; Cer et al., 2018; Zhang et al., 2018; Subramanian et al., 2018; Reimers & Gurevych, 2019; Zhang et al., 2020b).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “Learning universal sentence embeddings is a fundamental problem in natural language processing and has been studied extensively in the literature (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Logeswaran and Lee, 2018; Cer et al., 2018; Reimers and Gurevych, 2019, inter alia).”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “To establish baselines for CxC, we provide results for existing unimodal models for text (Bagof-Words, USE (Cer et al., 2018)) and images (InceptionV3 (Szegedy et al.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, neutral): “InferSent [25], Quick-Thought [77], and Universal Sentence Encoder [17].”
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks (2020, neutral): “, 2017) or USE (Cer et al., 2018).”
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2019, methodology): “Contextual-ized encoders can be pre-trained as unsupervised language models (Peters et al., 2018; Devlin et al., 2019), but they are usually improved on supervised transfer tasks such as Natural Language Inference (Cer et al., 2018).”
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (2021, methodology): “Furthermore, we experiment with the universal sentence encoder (USE) from Cer et al. (2018).”
Multilingual Universal Sentence Encoder for Semantic Retrieval (2019, methodology): “Similar to Cer et al. (2018) and Chidambaram et al. (2018), we target broad coverage using a multi-task dual-encoder training framework, with a single shared encoder supporting multiple down-stream tasks.”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, methodology): “The inputs are first tokenized using a pre-trained EfficientNet [58] (for images) and a pre-trained language model [59] (for the language instruction).”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, background): “Indeed, high-quality universal sentence embeddings have become an area of focus, and many supervised [4], semi-supervised [5, 6, 7] and unsupervised [8, 9, 10, 11, 12] approaches have been proposed.”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, neutral): “In fact, previous work has shown that their performance is sometimes even below static word embeddings and specialised sentence encoders (Cer et al., 2018) in lexical and sentence-level semantic similarity tasks (Reimers and Gurevych, 2019; Vulić et al.”
Text Data Augmentation for Deep Learning (2021, methodology): “Evaluated on the STS text similarity dataset, representations learned from DINO show improvements over state-of-the-art sentence embedding techniques trained with supervised learning, such as Universal Sentence Encoders [70] and Siamese BERT and RoBERTa models [71].”