All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

By William Timkey, Marten van Schijndel. 2021

176 citations recorded by the source corpus.

Similarity measures are a vital tool for understanding how language models represent and process language. Standard representational similarity measures such as cosine similarity and Euclidean distance have been successfully used in static word embedding models to understand how words cluster in semantic space. Recently, these measures have been applied to embeddings from contextualized models such as BERT and GPT-2. In this work, we call into question the informativity of such measures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate these measures. Moreover, we find a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model. We show that simple postprocessing techniques such as standardization are able to correct for rogue dimensions and reveal underlying representational quality. We argue that accounting for rogue dimensions is essential for any similarity-based analysis of contextual language models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “Outlier-dependent Quantization Through Proxy Quantization Outlier features that emerge in large language models (Gao et al., 2019; Timkey & van Schijndel, 2021; Bondarenko et al., 2021; Wei et al., 2022; Luo et al., 2021; Kovaleva et al., 2021; Puccetti et al., 2022) can cause large quantization…”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Large magnitude outlier features in language models have been studied before (Timkey and van Schijndel, 2021; Bondarenko et al., 2021; Wei et al., 2022; Luo et al., 2021).”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “Another issue with the logit lens is that transformer hidden states often contain a small number of very high variance dimensions, and these “rogue dimensions” (Timkey and van Schijndel, 2021) tend to be distributed unevenly across layers; see Figure 6 (top) for an example.”
A Simple and Effective Pruning Approach for Large Language Models (2023, result): “Our work is also related to recent studies on the existence of large magnitude outlier features in Transformer-based language models [5, 45, 52, 61, 71].”