Towards Better Decoding and Language Model Integration in Sequence to Sequence Models

By Jan Chorowski, Navdeep Jaitly. 2016

384 citations recorded by the source corpus.

The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion. In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters. We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used. We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

GLTR: Statistical Detection and Visualization of Generated Text (2019, background): “, 2018), beam search (Chorowski and Jaitly, 2016; Shao et al., 2017), temperature-modulated sampling (Dagan and Engelson, 1995), or even implicitly with rule-based templated approaches.”
Hierarchical Neural Story Generation (2018, methodology): “…together without architectural modiﬁcations: Ramachandran et al. (2016) use language models to initialize the encoder and decoder side of the seq2seq model independently, and Chorowski and Jaitly (2016) combine the predictions of the language model and seq2seq model solely at inference time.”
Unsupervised Speech Recognition (2021, methodology): “we can observe that this is equivalent to the formulation of language model fusion for end-toend systems that combines posterior and prior probabilities of text for decoding (Chorowski and Jaitly, 2017).”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “The most common method for leveraging text-only data is to train a language model (LM) and integrate it into the recognition process using shallow [17], cold [18], or deep fusion [19].”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “We also use a coverage penalty c [29].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We introduce a coverage penalty term [32] in the fusion score with parameter c for LAS models [33], while we use a constant non-blank reward ρ to shift the logits produced by the LM for RNN-T [34] networks [35, 36].”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “The RNN-based NMT approach, or RNMT, was quickly established as the de-facto standard for NMT, and gained rapid adoption into large-scale systems in industry, e.g. Baidu (Zhou et al., 2016), Google (Wu et al., 2016), and Systran (Crego et al., 2016).”
Self-Training for End-to-End Speech Recognition (2019, background): “Sequence-to-sequence models are known to easily fail at inference in two ways: looping and early stopping [14].”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, background): “Label smoothing [14, 16] is a regularization mechanism to prevent the model from making over-confident predictions.”
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions (2019, background): “Sequence-to-sequence beam search decoders are known to be unstable sometimes exhibiting worse performance with an increasing beam size [14].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “We compute either a PER (without fusing additional word/character-based LM (Chorowski and Jaitly, 2016) or imposing lexical constraints), or a CER after the LM (we use a CER instead of a WER to give partial credit).”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “…thank Jan Chorowski, Mark Daoust, Pietro Kreitlon Carolino, Dieterich Lawson, Navdeep Jaitly, George Tucker, Quoc V. Le, Kelvin Xu, Cinjon Resnick, Melody Guan, Matthew D. Hoffman, Jeffrey Dean, Kevin Swersky, Ashish Vaswani, and members of the Google Brain team for helpful discussions and insight.”
Letter-Based Speech Recognition with Gated ConvNets (2017, neutral): “3 (Chorowski and Jaitly, 2016) 6.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, methodology): “Neural Network Architecture The proposal model ry|x and the online model qy|x are sequence-to-sequence neural networks (Bahdanau et al., 2016; Chorowski & Jaitly, 2017) with the same time-depth separable (TDS) architecture proposed in Hannun et al. (2019).”
Cold Fusion: Training Seq2Seq Models Together with Language Models (2017, methodology): “The standard way to integrate language models is to train the Seq2Seq model and the language model independently and then combine their outputs to guide beam search [5, 6, 7].”
Specaugment on Large Scale Datasets (2019, methodology): “We use shallow fusion [22] with an LSTM language model (LM) with two fusion parameters— the LM weight and coverage penalty [23].”