Transformer-Based Acoustic Modeling for Hybrid Speech Recognition

By Yongqiang Wang, Abdel-rahman Mohamed, Duc Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang and 3 more. 2019

266 citations recorded by the source corpus.

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “Transformers [34] chenones 4gram word 2.”
Iterative Pseudo-Labeling for Speech Recognition (2020, positive): “Some of the most promising are from the Transformer family [1, 2, 3, 4, 5], which give state-of-the-art results on many ASR benchmarks and close the gap between end-to-end and hybrid systems.”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “This is a big improvement over previous CNN based architectures such as QuartzNet[4], and it outperforms transformer and LSTM based models [11, 10, 24, 9, 25].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “We replace forward/backward LSTM with Transformer, due to its superiority in modeling long context [30, 31].”
Contrastive Semi-Supervised Learning for ASR (2021, background): “Fenc has a VGG front-end and 12 transformer blocks (dim=768, FFN=3072, 8 heads, stride 4) following [27] ( 90M parameters).”
End-to-End Speech Recognition: A Survey (2023, result): “10, on Librispeech, the current best-published classical hybrid systems range around 2.3% (test-clean) and 4.9% (test-other) word error rate [222], [323], while there already are a number of E2E systems providing similar performance [205], [206], [224], [320], with some E2E systems clearly…”
Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset (2020, methodology): “In the past several years, the transformer model [17] has proven to present signiﬁcant performance improvement over LSTM in a range of tasks [30, 31].”
Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition (2019, background): “Recently, transformer or self-attention based models have been proven very effective in both hybrid and end-to-end speech recognition systems [24–27].”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, background): “Transformers [29, 30, 31, 32], may lead to better absolute WER numbers, we believe that the conclusions reached in our experiments are invariant to such modeling decisions.”
Scaling Up Online Speech Recognition Using ConvNets (2020, background): “Bi-directional RNNs and Transformers either require considerable changes for lowlatency deployment [11, 12] or degrade rapidly when limiting the amount of future context [8].”
A Better and Faster end-to-end Model for Streaming ASR (2020, methodology): “Instead, we reuse the existing convolution module that aggregates information from neighboring context to implicitly provide relative positional information [24, 25].”
Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, methodology): “We follow [6, 10] to use context-and position-dependent graphemes (i.e., chenones ) for CE baselines and CTC experiments.”
TransVG: End-to-End Visual Grounding with Transformers (2021, background): “This superiority has attracted a surge of research interest in applications of transformers in NLP tasks [11, 12, 39, 66] and speech recognition [33, 50].”
Triple M: A Practical Text-to-Speech Synthesis System with Multi-Guidance Attention and Multi-Band Multi-Time LPCNet (2021, methodology): “html The word error rate (WER) measured by an ASR system described in [21] was 4.”
Semantic Mask for Transformer based End-to-End Speech Recognition (2019, methodology): “Unlike [18] and [20], we use the same checkpoint for test-clean and test-other dataset.”
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, methodology): “Several ways to encode relative positions for Transformer-based speech recognition are compared in [51].”