A Tensorized Transformer for Language Modeling

By Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, M. Zhou. 2019

200 citations recorded by the source corpus.

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Subfields: Artificial Intelligence, Computational Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

Citations in the corpus

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, background): “From a different angle, [20, 29] have probed the architectural change of self-attention layer to make the Transformer lightweight.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “To alleviate this problem, various methods are proposed to compress these models, like using low-rank approximation (Ma et al., 2019; Lan et al., 2020), weight-sharing (Dehghani et al.”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some attempts to compress and accelerate inference of the Transformer-based models using low-rank approximation [15, 12], weight-sharing [4, 12], knowledge distillation [21, 23, 10, 28], quantization [1, 33, 22, 8] and pruning [16, 17, 3, 25].”
Encoding word order in complex embeddings (2019, unknown): “Method BLEU AED (Bahdanau et al., 2014) 26.8 AED+Linguistic (Sennrich & Haddow, 2016) (cid:63) 28.4 AED+BPE (Sennrich et al., 2016) 34.2 Transformer (Ma et al., 2019) 34 .”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “…works dedicated to improving the inference efficiency of PLMs, including pruning (Wang et al., 2020; Gordon et al., 2020), low-rank factorization (Ma et al., 2019; Noach and Goldberg, 2020; Lan et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Kim et al., 2021), knowledge…”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, methodology): “We use a three-layer tensorized transformer core-1 for PTB and a six-layer tensorized transformer core-1 for Wikitext-103, following [29].”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, unknown): “For instance, low-rank approximation Ma et al. (2019), Lan et al. (2019), weight-sharing Dehghani et al. (2018), Lan et al. (2019); knowledge distillation Sanh et al. (2019),Sun et al. (2019), Jiao et al. (2019); pruning Michel et al. (2019), Voita et al. (2019), Fan et al. (2019); dynamic network…”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “Tensorized Transformer [136] compresses the multi-head attention in Transformer.”
PowerNorm: Rethinking Batch Normalization in Transformers (2020, methodology): “Language Modeling We report the LM results in Table 2, using the tensorized transformer proposed in (Ma et al., 2019).”
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering (2020, methodology): “dup. This is an orthogonal approach that can be combined with our decomposition idea. However, for the paired-input tasks we consider, pruning heads only provides limited speedup. In more recent work Ma et al. (2019) propose approximating the quadratic attention computation with a tensor decompositi”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, methodology): “Beyond these methodologies, one orthogonal direction is to directly optimize the Transformer layer itself [29]–[31], which has not been paid enough attention and is the focus of this paper.”