Star-Transformer

By Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, Zheng Zhang. 2019

292 citations recorded by the source corpus.

Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving the capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ETC: Encoding Long and Structured Inputs in Transformers (2020, background): “Conceptually, the l2l attention piece is reshaped into a n l × (2 r + 1) matrix as illustrated in Figure 2c. 4 If r = 1 and n g = 1 , we recover exactly the Star Transformer (Section 2).”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, background): “Various modifications have been proposed to improve the efficiency of its multi-head attention and feedforward sublayers [2, 3].”
Blockwise Self-Attention for Long Document Understanding (2019, methodology): “This technique has been widely used in Transformer training (Ott et al., 2019; Liu et al., 2019).”
Sparse Sinkhorn Attention (2020, methodology): “Star Transformer (Guo et al., 2019) performs attention spar-siﬁcation by converting the dense graph into a starshaped topology using a shared relay node.”
Improving Transformer Models by Reordering their Sublayers (2019, background): “Transformer Modiﬁcations Unlike recent papers that tried to improve the transformer by modifying the sublayers such as So et al. (2019); Guo et al. (2019); Zhang et al. (2019); Correia et al. (2019), in this paper we do not modify the sublayers at all, but simply rear-range their order.”
ETC: Encoding Long and Structured Data in Transformers (2020, background): “Conceptually, the l2l attention piece is reshaped into a nl × (2k + 1) matrix as illustrated in Figure 2c.3 Notice that if we take k = 1 and ng = 1, we recover exactly the Star Transformer described in Section 2.”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, background): “Therefore, Transformer usually performs better on huge datasets and is easy to overﬁt on small datasets (Guo et al., 2019).”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, methodology): “We implemented our system using MxNet 1.5 (Chen et al., 2015) and GluonNLP 0.8.1 (Guo et al., 2019a) on a machine with 8 NVIDIA Tesla V100 GPUs, each with 16GB of memory.”
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, background): “…modeling has become a core task in the ﬁeld of natural language processing (NLP) such as classiﬁcation (Zhang et al., 2015), sequence tagging (Guo et al., 2019a), question answering (He and Golub, 2016), and scene text recognition (Baek et al., 2019; Hwang and Sung, 2016), with its…”
Efficient Transformers: A Survey (2020, methodology): “These models incorporate the notion of a global model memory, which is reminiscent of the Set Transformer’s inducing point method or the global model memory of the Star Transformer.”
DA-Transformer: Distance-aware Transformer (2020, background): “A core component of Transformer is multi-head self-attention, which is responsible for modeling the relations between contexts (Yang et al., 2019; Guo et al., 2019).”
TENER: Adapting Transformer Encoder for Named Entity Recognition (2019, methodology): “Although performance of the Transformer is higher than (Guo et al., 2019), it still lags behind the BiLSTM-based models (Ma and Hovy, 2016).”