Attention Is All You Need In Speech Separation

By Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, Jianyuan Zhong. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2020

785 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) have long been the dominant architecture in sequence-to-sequence learning. RNNs, however, are inherently sequential models that do not allow parallelization of their computations. Transformers are emerging as a natural alternative to standard RNNs, replacing recurrent computations with a multi-head attention mechanism.In this paper, we propose the SepFormer, a novel RNN-free Transformer-based neural network for speech separation. The Sep-Former learns short and long-term dependencies with a multi-scale approach that employs transformers. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets. It reaches an SI-SNRi of 22.3 dB on WSJ0-2mix and an SI-SNRi of 19.5 dB on WSJ0-3mix. The SepFormer inherits the parallelization advantages of Transformers and achieves a competitive performance even when downsampling the encoded representation by a factor of 8. It is thus significantly faster and it is less memory-demanding than the latest speech separation systems with comparable performance.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/51c9d4d2f50ac5707c1f889aa97f08350d549132

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “Moreover, it supports the recently proposed SepFormer [45], which uses a pipeline of two transformers within a dual-path framework.”
Generative Pre-training for Speech with Flow Matching (2023, methodology): “It is worth noting that the fine-tuning method presented here is a vanilla solution that might not scale well as the number of speakers increases, a more dedicated fine-tuning method is left as future work.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “Sepformer uses a binary mask prediction scheme for training while this masking network captures both short and long-term dependencies and provide higher accuracy in performance (Subakan et al., 2021).”
Hybrid Transformers for Music Source Separation (2022, methodology): “Band-Split RNN introduced a novel unsupervised augmentation technique requiring only mixes to improve its performance by 0.7 dB of SDR. Transformers have been used for speech source separation with SepFormer [20], which is similar to Dual-Path RNN: short range attention layers are interleaved with l”
A Practical Survey on Faster and Lighter Transformers (2021, background): “The self-attention’s main advantage is its ability to relate any two positions of the input sequence regardless of their distance, thus increasing performance signiicantly on a wide range of tasks, including natural language processing (NLP) [10, 24, 121], computer vision [12, 27, 57], speech recogn”