Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

By Sefik Emre Eskimez, Xiaofei Wang, Min Tang, Hemin Yang, Zirun Zhu, Zhuo Chen, Huaming Wang, Takuya Yoshioka. 2021

28 citations recorded by the source corpus.

With the surge of online meetings, it has become more critical than ever to provide high-quality speech audio and live captioning under various noise conditions. However, most monaural speech enhancement (SE) models introduce processing artifacts and thus degrade the performance of downstream tasks, including automatic speech recognition (ASR). This paper proposes a multi-task training framework to make the SE models unharmful to ASR. Because most ASR training samples do not have corresponding clean signal references, we alternately perform two model update steps called SE-step and ASR-step. The SE-step uses clean and noisy signal pairs and a signal-based loss function. The ASR-step applies a pre-trained ASR model to training signals enhanced with the SE model. A cross-entropy loss between the ASR output and reference transcriptions is calculated to update the SE model parameters. Experimental results with realistic large-scale settings using ASR models trained on 75,000-hour data show that the proposed framework improves the word error rate for the SE output by 11.82% with little compromise in the SE quality. Performance analysis is also carried out by changing the ASR model, the data used for the ASR-step, and the schedule of the two update steps.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Zhuo Chen
3. Takuya Yoshioka
4. Shujie Liu
5. Sanyuan Chen
6. Dong Yu
7. Naoyuki Kanda
8. Huaming Wang

In Electrical and Electronic Engineering

1. Zhuo Chen
2. Jinyu Li
3. Sanyuan Chen
4. Takuya Yoshioka

In Signal Processing

1. Jinyu Li
2. Zhuo Chen
3. Takuya Yoshioka
4. Shujie Liu
5. Sanyuan Chen
6. Naoyuki Kanda
7. Dong Yu
8. Huaming Wang

Citations in the corpus

• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “Our training data for DCCRN came from Microsoft’s internal dataset, and we further fine-tuned the model using the ASR objective based on the training recipe of [40].”
• Personalized speech enhancement: new models and Comprehensive evaluation (2021, methodology): “This was inspired by [5], which applied the MT training to the unconditional SE and showed superior ASR performance.”