All-but-the-Top: Simple and Effective Postprocessing for Word Representations

By Jiaqi Mu, Suma Bhat, Pramod Viswanath. 2017

407 citations recorded by the source corpus.

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a {\em very simple}, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations {\em even stronger}. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and { text classification}) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Primer in BERTology: What We Know About How BERT Works (2020, background): “Because isotropy was shown to be beneficial for static word embeddings (Mu and Viswanath, 2018), this might be a fruitful direction to explore for BERT.”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “…method is that it provides an effective mechanism to encourage diversity of word embedding vectors, which is widely observed to yield better generalization performance in neural language models (e.g., Mu et al., 2018; Gao et al., 2019; Liu et al., 2018b; Cogswell et al., 2016; Khodak et al., 2018).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “in analyzing word embeddings (Mu and Viswanath, 2018; Gao et al., 2019; Wang et al., 2020), and show that the contrastive objective can “flatten” the singular value distribution of sentence embeddings and make the representations more isotropic.”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “Recent research (Mu et al., 2018; Ethayarajh, 2019) analyzed contextualized embeddings in pretrained models and showed that the repre-”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, positive): “We also leverage a quantitative isotropy score (IS), proposed in prior work (Arora et al., 2016; Mu and Viswanath, 2018), and defined as follows:”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, methodology): “In this section we outline three such methods: standardization, all-but-the-top (Mu and Viswanath, 2018), and ranking (via Spearman correlation).”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, methodology): “Ex- tensions of the mathematical formulation, such as the use of word weighting when building context vectors as in Arora et al. (2018b) or of spectral information along the lines of Mu and Viswanath (2018), are also worthy of further study.”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, methodology): “…STS (Perone et al., 2018), and BERT over ELMo (Peters et al., 2018) due to better performance on linguistic probing tasks (Liu et al., 2019a After weighting, we perform top-3 principal component removal on the embedding matrices, which is beneﬁcial for word-level similarity tasks (Mu et al., 2018).”
Augmenting NLP models using Latent Feature Interpolations (2020, methodology): “Second, we take the difference of text energies (Mu et al., 2017) of the samples into consideration analogous to how Tokozume et al. (2017) takes into account the energy of the speech samples, in order to make the perception of the mixed sample xi : xj = λ : (1 - λ).”
Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation (2020, background): “For example, Gong et al. (2018); Mu and Viswanath (2018) show that word frequency significantly impact the geometry of word embeddings.”
Representation Degeneration Problem in Training Natural Language Generation Models (2019, result): “, 2014) reported in Mu et al. (2018) are also consistent with ours.”
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference (2020, background): “For example, in [37], a principal component analysis is performed on word embeddings to isolate the salient relationships between words.”
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks (2020, result): “While some papers [Arora et al., 2016, Mu and Viswanath, 2018] have noted that word embeddings are fairly random-like in the bulk to argue that the log partition function is constant for all (cid:107) θ (cid:107) 2 = 1 , our quadratic assumption is a bit stronger.”