Universal Sentence Encoder

By Daniel Matthew Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar and 3 more. 2018

2,093 citations recorded by the source corpus.

We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “Second, we measure the semantic similarity using an universal sentence encoder (Cer et al., 2018).”
Transfer Learning in Natural Language Processing (2019, background): “16 but not least, we will highlight the use of multitask learning for pretraining (Subramanian et al., 2018; Cer et al., 2018; Devlin et al., 2018).”
ERNIE: Enhanced Representation through Knowledge Integration (2019, methodology): “As shown in ﬁgure 3, our method introduces dialogue embedding to identify the roles in the dialogue, which is different from that of universal sentence encoder (Cer et al., 2018).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “…neural network architectures and training strategies (Rocktäschel et al., 2016; Gong et al., 2018; Peters et al., 2018; Wang et al., 2018), as well as to train effective, reusable sentence representations (Conneau et al., 2017; Subramanian et al., 2018; Cer et al., 2018; Conneau et al., 2018a).”
Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, methodology): “The encoded candidates are ﬂattened into vectors using the normalization from Cer et al. (2018) to produce an attention prediction over the memory.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…with the bot preﬁx) can be optimized by the model itself, but the remaining four rewards include eliciting positive responses from a human user or measuring user-bot response similarity (e.g. using word overlap or similarity in Universal Sentence Encoder (USE) embeddings (Cer et al., 2018)).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “To condition the policies on language, we utilize a pre-trained large sentence encoder language model [15].”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “In this section, we compare SBERT to average GloVe embeddings, InferSent (Conneau et al., 2017), and Universal Sentence Encoder (Cer et al., 2018).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, positive): “This was later extended to multitask learning, combining different training objectives like that of skip-thought, NLI, andmachine translation (Cer et al., 2018; Subramanian et al., 2018).”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “Although recently proposed sentence encoders have achieved remarkable re-sults (e.g., (Yin and Sch¨utze, 2015; Arora et al., 2017; Cer et al., 2018; Pagliardini et al., 2018)), most, if not all, of them are trained on a priori ﬁxed corpora.”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, methodology): “Related Works Cer et al. (2018) study the two-tower Transformer model as a universal sentence encoder.”
SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter (2019, methodology): “In particular, most of the systems adopted models known to be particularly suitable for dealing with texts, from Recurrent Neural Networks to recently proposed language models (Sabour et al., 2017; Cer et al., 2018).”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, background): “Multilingual Universal Sentence Encoder (mUSE) (Chidambaram et al., 2018; Yang et al., 2019) was trained in a multi-task setup on SNLI (Bowman et al., 2015) and on over a billion question-answer pairs from popular online forums and QA websites.”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “(3) A Multitask model (DET2T+I2T) is trained on a combination of tasks (Chidambaram et al., 2019).”
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, methodology): “Finally, Universal Sentence Encoder9 (Cer et al., 2018) is trained using multitask learning, whereby a single encoding model is used to feed multiple downstream tasks.”
Information Leakage in Embedding Models (2020, methodology): “A popular architecture for unsupervised sentence embedding is the dual-encoder model proposed in many prior works [5, 7, 26, 39, 40, 56, 72].”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, background): “a number of tasks including sentence classification (Iyyer et al., 2015; Cer et al., 2018), conversation input-response prediction (Yang et al.”
On Accurate Evaluation of GANs for Language Generation (2018, methodology): “To test robustness of FD to the choice of embedding model we use two additional sequence encoders [8] on the same data.”