Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model

By Sungrae Park, Geewook Kim, Junyeop Lee, Junbum Cha, Ji-Hoon Kim, Hwalsuk Lee. 2020

3 citations recorded by the source corpus.

This paper introduces a method that efficiently reduces the computational cost and parameter size of Transformer. The proposed model, refer to as Group-Transformer, splits feature space into multiple groups, factorizes the calculation paths, and reduces computations for the group interaction. Extensive experiments on two benchmark tasks, enwik8 and text8, prove our model’s effectiveness and efficiency in small-scale Transformers. To the best of our knowledge, Group-Transformer is the first attempt to design Transformer with the group strategy, widely used for efficient CNN architectures.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, methodology): “For summarization we employ Rouge score, which we know is a ﬂawed evaluation that does not tell the whole story; for that reason, it is difﬁcult to be certain our trade-offs are correct.”