Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer

By Sanyuan Chen, Yu Wu, Zhuo Chen, Takuya Yoshioka, Shujie Liu, Jinyu Li. 2020

35 citations recorded by the source corpus.

With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently. However, multi-channel speech separation sometimes does not necessarily need such a heavy structure for all time frames especially when the cross-talker challenge happens only occasionally. For example, in conversation scenarios, most regions contain only a single active speaker, where the separation task downgrades to a single speaker enhancement problem. It turns out that using a very deep network structure for dealing with signals with a low overlap ratio not only negatively affects the inference efficiency but also hurts the separation performance. To deal with this problem, we propose an early exit mechanism, which enables the Transformer model to handle different cases with adaptive depth. Experimental results indicate that not only does the early exit mechanism accelerate the inference, but it also improves the accuracy.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Zhuo Chen
4. Yu Wu
5. Takuya Yoshioka
6. Sanyuan Chen
7. Dong Yu
8. Furu Wei

In Signal Processing

1. Jinyu Li
2. Zhuo Chen
3. Takuya Yoshioka
4. Shujie Liu
5. Yu Wu
6. Sanyuan Chen
7. Dong Yu
8. Naoyuki Kanda

Citations in the corpus

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “We follow the previous work [22], [50], [77], [78] to evaluate our model with an end-to-end Transformer based ASR models [69], which achieves 2.08% and 4.95% word error rates (WERs) for LibriSpeech test-clean and test-other, respectively.”
• AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario (2021, background): “Therefore, most of the previous works on meeting transcription or relevant tasks such as overlapped speech recognition employ synthetic datasets for experimentation [4, 5, 6].”
• Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “The multi-head self-attention module is implemented with relative position embedding as [14, 5, 15].”