Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition

By Tara N. Sainath, Ron J. Weiss, Kevin Wilson, Bo Li, Arun Narayanan, Ehsan Variani, Michiel Bacchiani, Izhak Shafran, Andrew W. Senior, Kean K. Chin and 2 more. 2017

213 citations recorded by the source corpus.

Multichannel automatic speech recognition (ASR) systems commonly separate speech enhancement, including localization, beamforming, and postfiltering, from acoustic modeling. In this paper, we perform multichannel enhancement jointly with acoustic modeling in a deep neural network framework. Inspired by beamforming, which leverages differences in the fine time structure of the signal at different microphones to filter energy arriving from different directions, we explore modeling the raw time-domain waveform directly. We introduce a neural network architecture, which performs multichannel filtering in the first layer of the network, and show that this network learns to be robust to varying target speaker direction of arrival, performing as well as a model that is given oracle knowledge of the true target speaker direction. Next, we show how performance can be improved by factoring the first layer to separate the multichannel spatial filtering operation from a single channel filterbank which computes a frequency decomposition. We also introduce an adaptive variant, which updates the spatial filter coefficients at each time frame based on the previous inputs. Finally, we demonstrate that these approaches can be implemented more efficiently in the frequency domain. Overall, we find that such multichannel neural networks give a relative word error rate improvement of more than 5% compared to a traditional beamforming-based multichannel ASR system and more than 10% compared to a single channel waveform model.

Subfields: Artificial Intelligence, Computational Mechanics, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Speech & Audio

In Signal Processing

Citations in the corpus

Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, background): “Training on noisy data and using simulated noisy utterances to augment clean training are widely used [5, 3, 4, 8].”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [58]–[60], both networks were jointly trained in a more end-to-end fashion by extending the aforementioned CLDNN to raw waveform.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, unknown): “We do not detail those papers as well, since we focus on SSL in real-world reverberant environments.”
A Survey of Deep Learning: Platforms, Applications and Emerging Research Trends (2018, methodology): “[112] applied a deep neural network framework to jointly perform multichannel enhancement and acoustic modeling for automatic speech recognition (ASR).”
End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation (2019, background): “Another pipeline for neural beamformers is to directly estimate the beamforming ﬁlters in either time domain or frequency domain [9–13].”
Generation of Large-Scale Simulated Utterances in Virtual Rooms to Train Deep-Neural Networks for Far-Field Speech Recognition in Google Home (2017, methodology): “The model performs multichannel processing and acoustic modeling jointly, and has been shown to provide comparable or better performance to traditional enhancement techniques like beamforming [23].”
Cracking the cocktail party problem by multi-beam deep attractor network (2017, background): “In [27], several pooling steps are required, which is not suitable for multi-talker scenario.”
Acoustic Modeling for Distant Multi-talker Speech Recognition with Single- and Multi-channel Branches (2019, background): “For example, some papers [27, 28] proposed a convolutional neural network (CNN)based multi-channel SE block that was jointly trained with a neural network-based AM.”
FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing (2019, background): “The ﬁrst category, which we refer to as the ﬁltering-based (FB) approach, aims at learning a set of beamforming ﬁlters to perform ﬁlter-and-sum (FaS) beamforming in either the time domain [3–5] or frequency domain [6–9].”
Speech Processing for Digital Home Assistants: Combining signal processing with deep-learning techniques (2019, methodology): “A close look at the filter coefficients learned in the initial layers of the network showed that, indeed, beamformer-like spatial filters could be identified and that frequency resolutions resembling the mel filter bank were found [34].”
Attention-based Neural Beamforming Layers for Multi-channel Speech Recognition (2021, background): “Some studies have proposed training beamforming jointly with acoustic modeling [7, 8].”
Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement (2021, methodology): “ASR Multi-task Training: There are prior studies that employ joint or multi-task training for the front-end and backend [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22].”
Deep Learning for Environmentally Robust Speech Recognition (2017, background): “The de facto standard metric to evaluate the performance of ASR systems is Word Error Rate (WER) or Word Accuracy Rate (WAR).”