Exploring neural transducers for end-to-end speech recognition

By Eric Battenberg, Jitong Chen, Rewon Child, Adam Coates, Yashesh Gaur Yi Li, Hairong Liu, Sanjeev Satheesh, Anuroop Sriram, Zhenyao Zhu. 2017

234 citations recorded by the source corpus.

In this work, we perform an empirical comparison among the CTC, RNN-Transducer, and attention-based Seq2Seq models for end-to-end speech recognition. We show that, without any language model, Seq2Seq and RNN-Transducer models both outperform the best reported CTC models with a language model, on the popular Hub5'00 benchmark. On our internal diverse dataset, these trends continue — RNN-Transducer models rescored with a language model after beam search outperform our best CTC models. These results simplify the speech recognition pipeline so that decoding can now be expressed purely as neural network operations. We also study how the choice of encoder architecture affects the performance of the three models — when all encoder layers are forward only, and when encoders downsample the input representation aggressively.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, methodology): “More model, attention mechanism, and loss function details can be found in Battenberg et al. (2017).”
Beyond human-level accuracy: computational challenges in deep learning (2019, methodology): “We investigate the hybrid attention speech model diagrammed in Figure 5 [8].”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “CTC models exhibit fast inference but relatively poor performance (Battenberg et al., 2017).”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, background): “Recent investigations have shown promising results by applying the same approach for speech recognition [5,7,30–33].”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, neutral): “5 RNN-T [37] Y 4-gram 8.”
Cold Fusion: Training Seq2Seq Models Together with Language Models (2017, methodology): “We used coverage and length penalties as described in [18] for improved performance during beam search.”
OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models (2018, methodology): “…models built around the encoder-decoder paradigm (Sutskever et al., 2014) have been successfully used for natural language processing (NLP) (Vaswani et al., 2017), image-captioning (Xu et al., 2015), and automatic speech recognition (ASR) (Chan et al., 2015; Battenberg et al., 2017).”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, background): “Sequence-to-sequence learning [2] based on encoder-decoder attention models [3] has become popular for both machine translation [4] and speech recognition [5, 6, 7, 8, 9].”
Recent progresses in deep learning based acoustic models (2017, methodology): “…function and the training process to remove the frame-independence assumption of CTC. Specif-6 ically, RNN transducer was shown very effective [78], [79] as it incorporates acoustic model with its encoder, language model with its prediction network, and decoding process with its joint network.”
Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition (2020, methodology): “This is especially true for a streaming RNN-T with a unidirectional encoder network which sees a significant accuracy degradation from an offline RNN-T with a bidirectional encoder network [23].”
Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset (2020, background): “Great progress has been made to automatic speech recognition (ASR) with end-to-end (E2E) models [1, 2, 3, 4, 5, 6, 7, 8].”
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition (2018, methodology): “They remove the unreasonable frame-independence assumption made by Hidden Markov Model (HMM) [9] and Connectionist Temporal Classification (CTC) [10] models, enabling themselves to learn an implicit language model and optimize WER more directly [11].”
End-to-end Speech Recognition Using Lattice-free MMI (2018, unknown): “Also note that in training the attention-based and RNN-Transducer models (which are substantially larger) [33], data augmentation with background noise has been applied.”
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices (2018, methodology): “The end-to-end models with neural networks include connectionist temporal classification (CTC)-trained recurrent neural networks (RNN) [1, 2], encoder-decoder architectures [3, 4], and RNN transducers [5, 6].”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, unknown): “End-to-end automatic speech recognition (ASR) tries to simplify the modeling aspect, the training pipeline, and the decoding algorithm [1–8].”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, background): “1 Introduction Deep neural networks are now the state-of-the-art in a variety of challenging tasks, ranging from object recognition to natural language processing and graph analysis [28, 3, 52, 43, 36].”
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation (2019, background): “The dense matrices 1https://github.com/facebookresearch/dlrm and vectors are processed by a series of FC, CNN, or RNN layers.”