Mmdenselstm: An Efficient Combination of Convolutional and Recurrent Neural Networks for Audio Source Separation

By Naoya Takahashi, Nabarun Goswami, Yuki Mitsufuji. 2018

149 citations recorded by the source corpus.

Deep neural networks have become an indispensable technique for audio source separation (SS). It was recently reported that a variant of CNN architecture called MM-DenseNet was successfully employed to solve the SS problem of estimating source amplitudes, and state-of-the-art results were obtained for DSD 100 dataset. To further enhance MMDenseNet, here we propose a novel architecture that integrates long short-term memory (LSTM) in multiple scales with skip connections to efficiently model long-term structures within an audio context. The experimental results show that the proposed method outperforms MMDenseNet, LSTM and a blend of the two networks. The number of parameters and processing time of the proposed model are significantly less than those for simple blending. Furthermore, the proposed method yields better results than those obtained using ideal binary masks for a singing voice separation task.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Music Source Separation in the Waveform Domain (2019, methodology): “We also selected MMDenseLSTM [Takahashi et al., 2018], a multi-band dense net with LSTMs at different scales of the encoder and decoder.”
Denoising Auto-Encoder with Recurrent Skip Connections and Residual Regression for Music Source Separation (2018, methodology): “The result of our model is only second to that of the MMDenseNet [18] and MMDenseLSTM [19] models proposed by Takahashi et al.”
Singing Voice Separation: A Study on Training Data (2019, unknown): “In the previous literature [10, 11, 12, 13, 9] different architectures are usually compared using the same train/test datasets, but to the best of our knowledge, there are no previous works that study particularly the influence of these datasets.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, methodology): “adapted the MMDenseLSTM architecture, originally proposed in [178] for sound source separation, to the SELD problem.”
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (2020, background): “Hence, early studies only focus on magnitude related training target while ignoring phase [15–17], resynthesizing the estimated speech by simply applying estimated magnitude with the noisy speech phase.”
Meta-Learning Extractors for Music Source Separation (2020, methodology): “Here SDR was either taken from the respective papers ([10][11][12]) or from the SiSEC18 [4] evaluation scores ([13][14][15]), and we show the median of frames, median of tracks, evaluated with BSSEval v4 [4].”
Improving singing voice separation using Deep U-Net and Wave-U-Net with data augmentation (2019, background): “The current stateof-the-art, and winner of SiSEC 2018 [5], is a combination Long-Short Time Memory networks and Dense Convolutional Neural Networks, presented in [6] and use stereo signals.”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “Another method is to estimate magnitude and phase using two separate neural network modules which serially estimate magnitude and phase (Afouras et al., 2018; Takahashi et al., 2018a).”
The 2018 Signal Separation Evaluation Campaign (2018, unknown): “They are: UHL3 [28], TAK2 [26], TAK3 [27], TAU [27, 28].”