Deep Quaternion Networks

By Chase J. Gaudet, Anthony S. Maida. 2017

196 citations recorded by the source corpus.

The field of deep learning has seen significant advancement in recent years. However, much of the existing work has been focused on real-valued numbers. Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart. In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks. We develop the theoretical basis by reviewing quaternion convolutions, developing a novel quaternion weight initialization scheme, and developing novel algorithms for quaternion batch-normalization. These pieces are tested in a classification model by end-to-end training on the CIFAR −10 and CIFAR −100 data sets and a segmentation model by end-to-end training on the KITTI Road Segmentation data set. These quaternion networks show improved convergence compared to real-valued and complex-valued networks, especially on the segmentation task, while having fewer parameters.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Yi Tay
2. Georges Linarès
3. Shuai Zhang
4. R. Mori
5. Yoshua Bengio
6. Mohamed Morchid
7. Frank No'e
8. Marco Bertolini

In Computer Vision and Pattern Recognition

1. Yi Tay
2. Anh Tuan Luu
3. Shuohang Wang
4. Siu Cheung Hui
5. Qi Liu
6. Luc Van Gool
7. Mirco Ravanelli
8. Yoshua Bengio

Citations in the corpus

• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Hypercomplex representations Deep learning advances in the hypercomplex domain are in a nascent stage, and most work is fairly recent [40, 41, 42, 43, 44].”
• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, methodology): “Gaudet and Maida proposed deep Quaternion networks for image clas-sification, introducing basic tools such as Quater-nion batch normalization or Quaternion initialization (Gaudet and Maida, 2017).”
• Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters (2021, neutral): “(Gaudet & Maida, 2017; Parcollet et al., 2018a;b; Zhu et al., 2018; Tay et al., 2019).”
• Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition (2018, methodology): “A quaternion deep convolutional and residual neural network proposed in [19] have shown impressive results on the CIFAR images classification task.”
• Quaternion Knowledge Graph Embeddings (2019, methodology): “Quaternion representations are also useful for enhancing the performance of convolutional neural networks on multiple tasks such as automatic speech recognition [Parcollet et al.] and image classification [Gaudet and Maida, 2018, Parcollet et al., 2018a].”
• Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part III (2021, methodology): “Complex- and hypercomplex-based neural networks have received increasing attention in several applications, from computer vision to natural language processing (NLP) tasks [40,11,33,32,43,39].”
• QuaterNet: A Quaternion-based Recurrent Model for Human Motion (2018, unknown): “As for quaternions in neural networks, [16] proposes a hyper-complex extension of complexvalued convolutional neural networks, and [30] presents a variation of resilient backpropagation in quaternionic domain.”
• Quaternion Recurrent Neural Networks (2018, methodology): “The quaternion algebra is ensured by manipulating matrices of real numbers [5].”
• Quaternion Convolutional Neural Networks (2018, background): “In [9], a deep quaternion network is proposed.”