Which transformer architecture fits my data? A vocabulary bottleneck in self-attention

By Noam Wies, Yoav Levine, Daniel Jannai, Amnon Shashua. 2021

23 citations recorded by the source corpus.

After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains. An obstacle for their deployment over new modalities is the architectural configuration: the optimal depth-to-width ratio has been shown to dramatically vary across data types (e.g., $10$x larger over images than over language). We theoretically predict the existence of an embedding rank bottleneck that limits the contribution of self-attention width to the Transformer expressivity. We thus directly tie the input vocabulary size and rank to the optimal depth-to-width ratio, since a small vocabulary size or rank dictates an added advantage of depth over width. We empirically demonstrate the existence of this bottleneck and its implications on the depth-to-width interplay of Transformer architectures, linking the architecture variability across domains to the often glossed-over usage of different vocabulary sizes or embedding ranks in different domains. As an additional benefit, our rank bottlenecking framework allows us to identify size redundancies of $25\%-50\%$ in leading NLP models such as ALBERT and T5.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Laws for Neural Machine Translation (2021, methodology): “Considering the combinatorial expansion of the search space for scaling each one [29, 26, 37], in this study we choose to vary only the total number of Transformer Blocks, while keeping the internal hyper-parameters intact across different scales.”
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks (2022, methodology): “Even though the small vocabulary size of our model may indicate that it is too shallow for its size (Wies et al., 2021), we still choose to use the standard Transformer-Base configuration.”