Quaternion Recurrent Neural Networks

By Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, C. Trabelsi, R. Mori, Yoshua Bengio. 2018

153 citations recorded by the source corpus.

Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN), alongside with a quaternion long-short term memory neural network (QLSTM), that take into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition. Finally, we show that QRNN and QLSTM reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and LSTMs to reach better results, leading to a more compact representation of the relevant information.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Signal Processing

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “Hypercomplex representations Deep learning advances in the hypercomplex domain are in a nascent stage, and most work is fairly recent [40, 41, 42, 43, 44].”
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, methodology): “While Quaternion connectionist architectures have been considered in various deep learning application areas such as speech recognition (Parcollet et al., 2018b), kinematics/human motion (Pavllo et al., 2018) and computer vision (Gaudet and Maida, 2017), our work is the ﬁrst hypercomplex inductive…”
Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters (2021, methodology): “Though parameter reusing by component-wise partitioning in Quaternion space has demonstrated success (Parcollet et al., 2018b; Zhu et al., 2018; Parcollet et al., 2019; Tay et al., 2019), one key problem is that hypercomplex space only exists at very few predeﬁned dimensionalities, such as 4D…”
Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part III (2021, methodology): “We initialized the component weight matrices as initially described in complex- and quaternion Neural Networks [40,33].”
SpeechBrain: A General-Purpose Speech Toolkit (2021, background): “The toolkit also supports complex [97] and quaternion neural networks [98].”
Quaternion Knowledge Graph Embeddings (2019, methodology): “Quaternion representations are also useful for enhancing the performance of convolutional neural networks on multiple tasks such as automatic speech recognition [Parcollet et al.] and image classiﬁcation [Gaudet and Maida, 2018, Parcollet et al., 2018a].”
HABERTOR: An Efficient and Effective Deep Hatespeech Detector (2020, methodology): “Recently, Quaternion representations have shown its benefits over Euclidean representations in many neural designs (Parcollet et al., 2019; Tay et al., 2019): (i) a Quaternion number consists of a real component and three imaginary components, encouraging a richer extent of expressiveness; and (ii)…”