Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
By Nils Reimers, Iryna Gurevych. 2019
19,155 citations recorded by the source corpus.
BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Percy Liang
- Jianfeng Gao
- Noah A. Smith
- Hannaneh Hajishirzi
In Representation & Self-Supervised Learning
- Hinrich Schütze
- Iryna Gurevych
- Graham Neubig
- Percy Liang
- Yoshua Bengio
- Jianfeng Gao
- Ivan Vulic
- Luke S. Zettlemoyer
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “…we discuss in §3.1.1, such as toxicity/bias classifiers (Dathathri et al., 2019; Krause et al., 2020; Liu et al., 2021a), factuality metrics (Kryscinski et al., 2020; Goyal and Durrett, 2020), or functions that quantify similarity between source and target (Reimers and Gurevych, 2019, 2020).”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Empirically, we observed that using the same encoder, such as in Xiong et al. (2020) and Reimers & Gurevych (2019), generally improves robustness in the context of zero-shot transfer or few-shot learning, while having no impact on other settings.”
- Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “Table 7 compares the performance of demonstrations using uniform sampling to selective sampling by SBERT.”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “In the simplest approach, ranking becomes the problem of approximate nearest neighbor (ANN) search based on some simple metric such as cosine similarity [10, 12, 13, 17, 19, 22–24, 35, 36, 40, 42].”
- Memory-assisted prompt editing to improve GPT-3 after deployment (2022, methodology): “For such cases, we use Sentence transformers (Reimers and Gurevych, 2019) to encode the query, and cosine similarity with a threshold of 0.9 to find a matching key.”
- When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “We first embed the free-form textual feedback and the immediate next bot reply in recorded conversations with Sentence-BERT (Reimers and Gurevych, 2019), and then calculate their cosine similarity score.”
- Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “Specifically, given an input sentence x and the set of labeled and pseudo-labeled data consists of ( x ′ , e ′ , y ′ ) , we retrieve N explanations E := { e ′ i } N i =1 of which the corresponding sentences x ′ are the most similar to the input sentence x , measured by the cosine similarity between t”
- Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “[20] prompt GPT-3 [9] and Codex [56] to generate action plans for embodied agents, where each action step is semantically translated to an admissible action with a Sentence-RoBERTa model [57, 58].”
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, methodology): “For Translation LM, we mainly use Sentence-RoBERTa-355M and provide relevant ablations in Section 5.3.”
- LILA: Language-Informed Latent Actions (2021, methodology): “To allow for out-of-the-box generalization to novel user utterances such as those that describe similar behaviors but with different words or phrases, we tap into the power of pretrained models [14, 15].”
- Generating Datasets with Pretrained Language Models (2021, methodology): “While pretrained language models (PLMs) achieve strong results for many NLP tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019), they do not produce good sentence embeddings out of the box (Reimers and Gurevych, 2019).”
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “However, there exist approaches [14, 53] that solve regression tasks by training on classification tasks.”
- WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, methodology): “We explore the semantic similarity between the premise and hypothesis within each label class using Sentence-BERT (Reimers and Gurevych, 2019); these distributions are shown in Figure 4.”
- Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “When multiple changes occur, we label it with the control code which most significantly changes the semantics of the corresponding subphrase as computed by SBERT (Reimers and Gurevych, 2019).”
- Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “…previous approaches including BERT (Mean) (Devlin et al., 2018), InferSent (Conneau et al., 2017), GenSen (Subramanian et al., 2018), USE (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), uSIF (Ethayarajh, 2018a), Charagram (Wieting and Gimpel, 2017) and BGT (Wieting et al., 2019b).”
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, positive): “We further find that in line with previous work (Phang et al., 2018; Conneau and Kiela, 2018; Reimers and Gurevych, 2019), MultiNLI is a useful intermediate task that benefits a large number of target tasks, e.g. BoolQ, SICK, CSQA, SST-2, CB, MRPC, RTE .”
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “To this end, we extract Sentence-BERT [36] representation (SeBERT) for the answers and find the furthest from the other four options as the correct answer for each question.”
- What Makes Good In-Context Examples for GPT-3? (2021, background): “(Reimers and Gurevych, 2019, 2020) have shown that these fine-tuned encoders have achieved great performance on tasks such as sentence clustering, para-phrase mining, and information retrieval.”
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, background): “The generated lectures and explanations are evaluated by automatic metrics [44, 28, 49] and human scores by annotators.”