Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

By Nils Reimers, Iryna Gurevych. 2019

19,155 citations recorded by the source corpus.

BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Percy Liang
  3. Luke S. Zettlemoyer
  4. Jianfeng Gao
  5. Yejin Choi
  6. Graham Neubig
  7. Noah A. Smith
  8. Caiming Xiong

In Natural Language Processing

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Yejin Choi
  4. Graham Neubig
  5. Percy Liang
  6. Jianfeng Gao
  7. Noah A. Smith
  8. Hannaneh Hajishirzi

In Representation & Self-Supervised Learning

  1. Hinrich Schütze
  2. Iryna Gurevych
  3. Graham Neubig
  4. Percy Liang
  5. Yoshua Bengio
  6. Jianfeng Gao
  7. Ivan Vulic
  8. Luke S. Zettlemoyer

Citations in the corpus