DeLighT: Deep and Light-weight Transformer

By Sachin Mehta, Marjan Ghazvininejad, Srini Iyer, Luke S. Zettlemoyer, Hannaneh Hajishirzi. 2021

99 citations recorded by the source corpus.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Mostafa Dehghani
2. Yi Tay
3. Yu Qiao
4. Jifeng Dai
5. Lucas Beyer
6. Sachin Mehta
7. Anurag Arnab
8. Mohammad Rastegari

In Efficiency & Compression

1. Yi Tay
2. Mostafa Dehghani
3. Yu Qiao
4. Sachin Mehta
5. Ji-rong Wen
6. Liang Sun
7. Tong Lu
8. Mohammad Rastegari

Citations in the corpus

• The Efficiency Misnomer (2021, background): “That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.”
• Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, background): “However, stacked self-attention layers involve a large number of parameters, which might lead to the over-parameterized architecture of Transformer-based methods [13, 35].”
• Transformers in Time Series: A Survey (2022, background): “…Transformer variants in NLP and CV which also have architecture-level model designs to fit different purposes, such as lightweight [Wu et al. , 2020b; Mehta et al. , 2021], cross-block connectivity [ Bapna et al. , 2018 ] , adaptive computation time [ Dehghani et al. , 2019; Xin et al. , 2020 ] ,…”
• Vision Transformer Adapter for Dense Predictions (2022, methodology): “Considering the efficiency, we refer [56] and set the ratio of CFFN to 1/4.”
• Separable Self-attention for Mobile Vision Transformers (2022, background): “Improving transformer-based models There has been significant work on improving the efficiency of transformers (Liu et al., 2021; Mehta & Rastegari, 2022; Mehta et al., 2021; Wu et al., 2021; Heo et al., 2021).”
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, result): “Because previous works (e.g., Howard et al., 2017; Mehta et al., 2021a) have shown that networks designed using these layers are heavy-weight, a natural question arises: Why MobileViT is light-weight?”