Limits to Depth Efficiencies of Self-Attention

By Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, Amnon Shashua. 2020

56 citations recorded by the source corpus.

Self-attention architectures, which are rapidly pushing the frontier in natural language processing, demonstrate a surprising depth-inefficient behavior: Empirical signals indicate that increasing the internal representation (network width) is just as useful as increasing the number of self-attention layers (network depth). In this paper, we theoretically study the interplay between depth and width in self-attention, and shed light on the root of the above phenomenon. We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention, and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width. Specifically, we pinpoint a "depth threshold" that is logarithmic in $d_x$, the network width: $L_{\textrm{th}}=\log_{3}(d_x)$. For networks of depth that is below the threshold, we establish a double-exponential depth-efficiency of the self-attention operation, while for depths over the threshold we show that depth-inefficiency kicks in. Our predictions strongly accord with extensive empirical ablations in Kaplan et al. (2020), accounting for the different behaviors in the two depth-(in)efficiency regimes. By identifying network width as a limiting factor, our analysis indicates that solutions for dramatically increasing the width can facilitate the next leap in self-attention expressivity.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Information Systems, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Information Systems

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Neural Machine Translation (2021, methodology): “Considering the combinatorial expansion of the search space for scaling each one [29, 26, 37], in this study we choose to vary only the total number of Transformer Blocks, while keeping the internal hyper-parameters intact across different scales.”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Levine et al. (2020) proposed a theoretically motivated and empirically backed law describing the optimal compromise between width and depth.”
A Comprehensive Overview of Large Language Models (2023, unknown): “Xuan Yuan 2.0 combined the pre-training and fine-tuning stages to avoid catastrophic forgetting.”
BloombergGPT: A Large Language Model for Finance (2023, methodology): “Comparing model architectures, Levine et al. (2020) studied the scaling of models that use self-attention and derived guidelines for depth-to-width allocation. Tay et al. (2021) reported that model shape (depth-width ratio) impacted performance on downstream tasks even if it had minimal impact on th”
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021, methodology): “In the next subsection we will show a corollary from (Levine et al., 2020) that uses this claim to prove the lower bound in theorem 2 of the main text.”