Deep Complex Networks

By C. Trabelsi, Olexa Bilaniuk, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, C. Pal. 2017

1,046 citations recorded by the source corpus.

At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their real-valued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Dong Yu
3. Yi Tay
4. Jinyu Li
5. Quoc V. Le
6. Mirco Ravanelli
7. Li Deng
8. Bo Li

In Signal Processing

1. Yoshua Bengio
2. Jinyu Li
3. Dong Yu
4. Mirco Ravanelli
5. Zhuo Chen
6. Li Deng
7. Tara N. Sainath
8. Shinji Watanabe

Citations in the corpus

• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, background): “Complex networks (Trabelsi et al., 2017), in general, have also demonstrated promise over real networks.”
• Encoding word order in complex embeddings (2019, methodology): “Prior work in NLP (Li et al., 2019), Information Retrieval (Van Rijsbergen, 2004) and Machine Learning (Trabelsi et al., 2017) has shown the usefulness of complex numbers as richer representations.”
• Invariances and Data Augmentation for Supervised Music Transcription (2017, methodology): “Recent developments in this area model the acoustics using deep neural networks [11, 12] or convolutional neural networks [13, 14, 7].”
• SpeechBrain: A General-Purpose Speech Toolkit (2021, background): “The toolkit also supports complex [97] and quaternion neural networks [98].”
• Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition (2018, methodology): “However, a specific quaternion is used for each RGB color value as in [14] rather than integrating pixel multiple views as in [37], and suggested in this paper for an ASR task.”
• Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement (2021, methodology): “They took advantages of the U-net structure and deep complex networks [10], which was shown to be useful for SE [11].”
• FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement (2022, neutral): “The complex BN and the LeakyReLU follow [14].”
• Deep Learning for Audio Signal Processing (2019, background): “(convolution, pooling, activation functions) in a DNN may be extended to the complex domain [64].”
• Classification with Quantum Neural Networks on Near Term Processors (2018, background): “Researchers in classical machine learning recently started to investigate the advantage of using unitary transformations to control gradient blow up [6][7][8][9].”
• Hierarchical quantum classifiers (2018, methodology): “As has been observed in the context of the timedependant variational principle applied to tensor networks, the use of complex weights often prevents optimization from getting stuck in local minima.(21,22) We also explored a number of other methods for parameterizing the unitaries; Fig.”
• Quaternion Recurrent Neural Networks (2018, methodology): “…a deep quaternion network (Parcollet et al., 2016; 2017a;b), a deep quaternion convolutional network (Gaudet & Maida, 2018; Parcollet et al., 2018), or a deep complex convolutional network (Trabelsi et al., 2017) have been employed for challenging tasks such as images and language processing.”
• Deep Denoising Neural Network Assisted Compressive Channel Estimation for mmWave Intelligent Reflecting Surfaces (2020, methodology): “More specifically, we integrate the complex building blocks inspired from [19] into the DnCNN, so that the denoiser can be tailored for complex signal processing.”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Additionally, using complex number shows potential to improve the expression of neural networks in recent years [93], so the explanation of a complex neuron with multiple inputs and a single output based on the QR decomposition [94] deserves more attention.”
• Quaternion Convolutional Neural Networks (2018, background): “It has been proven that deep complex networks can obtain competitive results with real-valued models on audio-related tasks [32].”
• Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “Details on batch normalization and weight initialization for complex networks can be found in Trabelsi et al. (2018).”