Single-Channel Multi-talker Speech Recognition with Permutation Invariant Training

By Yanmin Qian, Xuankai Chang, Dong Yu. 2017

87 citations recorded by the source corpus.

Although great progresses have been made in automatic speech recognition (ASR), significant performance degradation is still observed when recognizing multi-talker mixed speech. In this paper, we propose and evaluate several architectures to address this problem under the assumption that only a single channel of mixed signal is available. Our technique extends permutation invariant training (PIT) by introducing the front-end feature separation module with the minimum mean square error (MSE) criterion and the back-end recognition module with the minimum cross entropy (CE) criterion. More specifically, during training we compute the average MSE or CE over the whole utterance for each possible utterance-level output-target assignment, pick the one with the minimum MSE or CE, and optimize for that assignment. This strategy elegantly solves the label permutation problem observed in the deep learning based multi-talker mixed speech separation and recognition systems. The proposed architectures are evaluated and compared on an artificially mixed AMI dataset with both two- and three-talker mixed speech. The experimental results indicate that our proposed architectures can cut the word error rate (WER) by 45.0% and 25.0% relatively against the state-of-the-art single-talker speech recognition system across all speakers when their energies are comparable, for two- and three-talker mixed speech, respectively. To our knowledge, this is the first work on the multi-talker mixed speech recognition on the challenging speaker-independent spontaneous large vocabulary continuous speech task.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Single Channel Target Speaker Extraction and Recognition with Speaker Beam (2018, methodology): “Solving such mismatch between a speech enhancement front-end and an ASR back-end has been addressed by jointly training both modules [15, 21, 22].”
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals (2020, methodology): “conditional chain Transformer-based CTC with the other systems including the HMM systems (1): deep clustering (DPCL)+GMM-HMM [15], (2): PIT-DNN-HMM [28] , and (3): DPCL+DNN-HMM [26], and the attention-based systems (4): PIT-RNN [6] and (5): PIT-Transformer.”
Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR (2020, background): “Many efforts have already been put into the field of multitalker speech recognition [3–11].”