On the Sentence Embeddings from BERT for Semantic Textual Similarity

By Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, Lei Li. 2020

254 citations recorded by the source corpus.

Pre-trained contextual representations like BERT have achieved great success in natural language processing. However, the sentence embeddings from the pre-trained language models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We first reveal the theoretical connection between the masked language model pre-training objective and the semantic similarity task theoretically, and then analyze the BERT sentence embeddings empirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks. The code is available at https://github.com/bohanli/BERT-flow.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wayne Xin Zhao
2. Ji-rong Wen
3. Bo Li
4. Tara N. Sainath
5. Percy Liang
6. Trevor Strohman
7. Yupeng Hou
8. Jinyu Li

In Natural Language Processing

1. Bo Li
2. Ji-rong Wen
3. Wayne Xin Zhao
4. Tianyu Gao
5. James Y. Zou
6. Tara N. Sainath
7. Weixin Liang
8. Shuo-yiin Chang

In Representation & Self-Supervised Learning

1. Bo Li
2. James Y. Zou
3. Weixin Liang
4. Ji-rong Wen
5. Tara N. Sainath
6. Yangyiwen Ou
7. Xingcheng Yao
8. Yaliang Li

Citations in the corpus

• SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “…compare our reproduced results to reported results of SBERT and BERT-whitening, and find that Reimers and Gurevych (2019) take the “all” setting but Li et al. (2020); Su et al. (2021) take the “wmean” setting, even though Li et al. (2020) claim that they take the same setting as Reimers 12Li et…”
• Whitening Sentence Representations for Better Semantics and Faster Retrieval (2021, methodology): “(Li et al., 2020) verified that sentence embedding from BERT (Devlin et al., 2019) has included sufficient semantics although it is not exploited properly.”
• Context-Aware end-to-end ASR Using Self-Attentive Embedding and Tensor Fusion (2023, methodology): “While [CLS] has been commonly used as sentence embedding, it has been shown that using the BERT word embeddings are consistently better than the [CLS] embedding when serving as sentence embeddings [18, 19].”
• Towards Universal Sequence Representation Learning for Recommender Systems (2022, methodology): “Existing studies [11] have found that BERT induces a non-smooth anisotropic semantic space for general texts.”
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, methodology): “A common explanation is that the unbalanced distribution of word frequencies biased the optimization [15, 33].”