State-of-the-Art Speech Recognition with Sequence-to-Sequence Models

By Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Z. Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Katya Gonina and 4 more. 2017

1,190 citations recorded by the source corpus.

Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network. In previous work, we have shown that such architectures are comparable to state-of-the-art ASR systems on dictation tasks, but it was not clear if such architectures would be practical for more challenging tasks such as voice search. In this work, we explore a variety of structural and optimization improvements to our LAS model which significantly improve performance. On the structural side, we show that word piece models can be used instead of graphemes. We also introduce a multi-head attention architecture, which offers improvements over the commonly-used single-head attention. On the optimization side, we explore synchronous training, scheduled sampling, label smoothing, and minimum word error rate optimization, which are all shown to improve accuracy. We present results with a unidirectional LSTM encoder for streaming recognition. On a 12, 500 hour voice search task, we find that the proposed changes improve the WER from 9.2% to 5.6%, while the best conventional system achieves 6.7%; on a dictation task our model achieves a WER of 4.1% compared to 5% for the conventional system.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

A Simple Method for Commonsense Reasoning (2018, background): “Although deep neural networks have achieved remarkable successes (e.g., [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), their dependence on supervised learning has been challenged as a signiﬁcant weakness.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “For years, the fields have been continuously reporting new state of the art results using varieties of model architectures for computer vision tasks [57, 58, 7], for natural language understanding tasks [59, 60, 61], for speech recognition and synthesis tasks [62, 63, 64, 65, 66].”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, neutral): “Empirically, seq2seq often performs the best (Chiu et al., 2018).”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, background): “While many advances in end-to-end ASR come as the result of neural architecture search (Prabhavalkar et al., 2017; Zhou et al., 2018; Chiu et al., 2018b), we additionally show that simple semi-supervision via pseudo-labeling significantly bridges the performance gap between a variety of different…”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “For instance, end-to-end speech recognition models have achieved state-of-the-art (SoTA) results on LibriSpeech [5] and large proprietary datasets [6].”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Attention-based end-to-end (E2E) speech recognition systems map audio features directly to text-level representations [1, 2, 3, 4, 5].”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, background): “However, these models tend to overfit easily and require large amounts of training data [8].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Recurrent neural networks (RNNs) have been the defacto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human-level accuracy in many domains (Chiu et al., 2018).”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large labeled datasets and make little to no use of unsupervised representation learning.”
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions (2019, background): “Our work builds on a large body of work aimed at improving sequence-to-sequence models with attention for both speech recognition [5, 14, 23] and other application domains.”
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “AED models adopt 6 Transformer encoder and decoder layers, with additional cross-attention modules, for MT and Fbank-based ASR tasks.”
A Survey on Neural Speech Synthesis (2021, background): “However, for the tasks like automatic speech recognition [49, 34, 47] and text to speech synthesis [376], the alignments between text and speech have some specific properties.”
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, methodology): “thermore, the modeling units of WPM have been explored in [6], which are sub-word units ranging from graphemes all the way up to entire words.”
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese (2018, background): “Experts have shown significant interest in the area of sequenceto-sequence modeling with attention [1, 2, 3, 4] on ASR tasks in recent years.”
Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (2017, background): “Our previous work [32] has shown that LAS can approach state-of-the-art performance when trained on larger amounts of data.”
ESPnet: End-to-End Speech Processing Toolkit (2018, background): “Actually, [12] and [16] report comparable or superior performance to the state-of-the-art hybrid HMM/DNN systems in very large English tasks, although these results are not usually accomplished by many of research communities due to the lack of computational resources.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “, 2014; Kalchbrenner and Blunsom, 2013) and speech recognition (Chorowski et al., 2015; Chiu et al., 2017) have been made.”