Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks

By Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, Rongrong Ji. 2022

63 citations recorded by the source corpus.

Despite the exciting performance, Transformer is criticized for its excessive parameters and computation cost. However, compressing Transformer remains as an open problem due to its internal complexity of the layer designs, i.e., Multi-Head Attention (MHA) and Feed-Forward Network (FFN). To address this issue, we introduce Group-wise Transformation towards a universal yet lightweight Transformer for vision-and-language tasks, termed as LW-Transformer. LW-Transformer applies Group-wise Transformation to reduce both the parameters and computations of Transformer, while also preserving its two main properties, i.e., the efficient attention modeling on diverse subspaces of MHA, and the expanding-scaling feature transformation of FFN. We apply LW-Transformer to a set of Transformer-based networks, and quantitatively measure them on three vision-and-language tasks and six benchmark datasets. Experimental results show that while saving a large number of parameters and computations, LW-Transformer achieves very competitive performance against the original Transformer networks for vision-and-language tasks. To examine the generalization ability, we apply LW-Transformer to the task of image classification, and build its network based on a recently proposed image Transformer called Swin-Transformer, where the effectiveness can be also confirmed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. J. Ainslie
2. James Lee-Thorp
3. Sumit K. Sanghai
4. Han Hu
5. Xinyu Liu
6. Michiel de Jong
7. Yuqing Yang
8. Federico Lebr'on

In Computer Vision and Pattern Recognition

1. James Lee-Thorp
2. Sumit K. Sanghai
3. Han Hu
4. J. Ainslie
5. Xinyu Liu
6. Michiel de Jong
7. Yuqing Yang
8. Houwen Peng

In Natural Language Processing

1. James Lee-Thorp
2. J. Ainslie
3. Sumit K. Sanghai
4. Michiel de Jong
5. Houwen Peng
6. Yury Zemlyanskiy

Citations in the corpus

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, methodology): “Due to limited computation, we also do not compare our XXL GQA model to a comparitive model trained from scratch, so we do not know the relative performance of uptraining vs training from scratch.”
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, background): “Another line of work combines ViTs with lightweight CNNs to build efficient architectures [9, 47, 49, 50, 81].”