Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

By Yi Tay, Aston Zhang, Anh Tuan Luu, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, Siu Cheung Hui. 2019

71 citations recorded by the source corpus.

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly (75%) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to 75% reduction in parameter size without significant loss in performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Fei Tan
6. Serim Park
7. Yifan Hu
8. Frank No'e

In Computer Vision and Pattern Recognition

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Frank No'e
6. Marco Bertolini
7. Tuan Le
8. Geewook Kim

In Natural Language Processing

1. Donald Metzler
2. Yi Tay
3. Mostafa Dehghani
4. Dara Bahri
5. Frank No'e
6. Ji-Hoon Kim
7. Kyumin Lee
8. Djork-Arné Clevert

In Efficiency & Compression

1. Dara Bahri
2. Yi Tay
3. Donald Metzler
4. Mostafa Dehghani
5. Sungrae Park

Citations in the corpus

• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “Hypercomplex representations Deep learning advances in the hypercomplex domain are in a nascent stage, and most work is fairly recent [40, 41, 42, 43, 44].”
• Efficient Transformers: A Survey (2020, methodology): “On the other hand, the Quaternion Transformer (Tay et al., 2019) proposes a weight sharing scheme inspired by Hamilton products that locally shares the components in the linear transformation layers.”
• HABERTOR: An Efficient and Effective Deep Hatespeech Detector (2020, methodology): “Recently, Quaternion representations have shown its benefits over Euclidean representations in many neural designs (Parcollet et al., 2019; Tay et al., 2019): (i) a Quaternion number consists of a real component and three imaginary components, encouraging a richer extent of expressiveness; and (ii) ”
• Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, methodology): “(Tay et al., 2019) utilizes quaternion algebra to build lightweight modules for Transformer.”
• Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part III (2021, background): “Complex- and hypercomplex-based neural networks have received increasing attention in several applications, from computer vision to natural language processing (NLP) tasks [40,11,33,32,43,39].”