Whitening Sentence Representations for Better Semantics and Faster Retrieval

By Jianlin Su, Jiarun Cao, Weijie Liu, Yangyiwen Ou. arXiv.org, 2021

370 citations recorded by the source corpus.

Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. Previous work has shown that the anisotropy problem is an critical bottleneck for BERT-based sentence representation which hinders the model to fully utilize the underlying semantic features. Therefore, some attempts of boosting the isotropy of sentence distribution, such as flow-based model, have been applied to sentence representations and achieved some improvement. In this paper, we find that the whitening operation in traditional machine learning can similarly enhance the isotropy of sentence representations and achieve competitive results. Furthermore, the whitening technique is also capable of reducing the dimensionality of the sentence representation. Our experimental results show that it can not only achieve promising performance but also significantly reduce the storage cost and accelerate the model retrieval speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a2fd50aa4dff5e04ed8535d84550da8bff316208

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Pre-trained Language Models Can be Fully Zero-Shot Learners (2022, methodology): “As suggested in (Su et al., 2021), we whiten the contextualized output of RoBERTa by a linear transformation obtained from the contex-tualized embedding of all words in vocabulary.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, background): “To address this issue, subsequent studies have proposed different approaches, including supervised fine-tuning (Reimers and Gurevych, 2019), normalizing flow (Li et al., 2020), normalizing flow (Li et al., 2020), whitening (Su et al., 2021), or unsupervised contrastive learning (Gao et al., 2021).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “Another line of work focuses on regularizing embeddings (Li et al., 2020; Su et al., 2021; Huang et al., 2021) to alleviate the representation degeneration problem (as discussed in §5), and yields substantial improvement over pre-trained language models.”
Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders (2024, positive): “We use principal component analysis (PCA) with whitening as the default adaptor due to its simplicity and efficiency (Huang et al., 2021; Su et al., 2021).”
Positional Artefacts Propagate Through Masked Language Model Embeddings (2020, methodology): “Likewise, we would like to stress that our manual clipping operation was performed for illustrative purposes and that interested researchers should employ more systematic post-hoc normalization strategies, e.g. whitening (Su et al., 2021), when working with hidden states directly.”
Text and Code Embeddings by Contrastive Pre-Training (2022, background): “The whitening operation is another alternative operation to improve the isotropy of the embedding space (Su et al., 2021).”
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, methodology): “Post-processing methods [33, 43, 2, 35] or modified training objective [15, 47, 16] alleviate the cone effect and improve downstream performance.”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, neutral): “Sentence embeddings induced from MLMs without fine-tuning thus suffer from spatial anistropy (Li et al., 2020; Su et al., 2021).”
Towards Universal Sequence Representation Learning for Recommender Systems (2022, methodology): “Inspired by recent works on whitening-based methods [9, 21], we conduct a simple linear transformation to transform original BERT representations for deriving isotropic semantic representations.”
Effects of Model Choice, Corpus Context, and Post Hoc Correction on Layer-Level Embedding Degradation in Clinical Document Retrieval: Experimental Study (2026, unknown): “Subsequent work has explored isotropy-promoting training objectives (BioLORD-2023 [13] adopts an explicit isotropy term in its contrastive loss) and post hoc whitening transforms [14-16] that compress eigenspectra.”
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “In research on text embeddings, enhancing isotropy has been shown to improve performance on STS tasks (Mu and Viswanath, 2018; Li et al., 2020; Su et al., 2021; Huang et al., 2021; Yokoi et al., 2024).”
Length-Induced Embedding Collapse in PLM-based Models (2024, unknown): “In addition to TempScale, several post-processing methods have been proposed to address high similarity in embeddings, such as the Flow Function (Li et al., 2020) and Whitening (Su et al., 2021).”