An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model

By Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N. Sainath, Z. Chen, Rohit Prabhavalkar. 2017

282 citations recorded by the source corpus.

Attention-based sequence-to-sequence models for automatic speech recognition jointly train an acoustic model, language model, and alignment mechanism. Thus, the language model component is only trained on transcribed audio-text pairs. This leads to the use of shallow fusion with an external language model at inference time. Shallow fusion refers to log-linear interpolation with a separately trained language model at each step of the beam search. In this work, we investigate the behavior of shallow fusion across a range of conditions: different types of language models, different decoding units, and different tasks. On Google Voice Search, we demonstrate that the use of shallow fusion with an neural LM with wordpieces yields a 9.1% relative word error rate reduction (WERR) over our competitive attention-based sequence-to-sequence model, obviating the need for second-pass rescoring.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

In Signal Processing

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “The most common method for leveraging text-only data is to train a language model (LM) and integrate it into the recognition process using shallow [17], cold [18], or deep fusion [19].”
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, background): “Typically, word-level LMs have a much lower perplexity compared to grapheme-level LMs [20].”
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, background): “Similar to [17, 18], at the current frame t, these features are stacked with 3 frames to the left and downsampled to a 30ms frame rate.”
Streaming End-to-end Speech Recognition for Mobile Devices (2018, methodology): “In the E2E RNN-T model, we use a technique similar to [36], to compute biasing scores PC(y), which are interpolated with the base model P (y|x) using shallow-fusion [37] during beam search: y = argmax”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “• We combine this and a shallow integrated RNNLM [18, 19] to obtain a high performance on unpaired data.”
A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition (2018, methodology): “05 } , following [8].”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, background): “Therefore, shallow fusion shows performance gains in many ASR tasks [5, 18, 19].”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “For example, training an RNNLM on text-only data and fusing this into the E2E model, via techniques such as shallow, cold and deep fusion, has shown improvements on long-tail queries [10, 11, 12].”
Deep Audio-Visual Speech Recognition (2018, methodology): “Decoding for both models is performed with a left-to-right beam search where the LM log-probabilities are combined with the model’s outputs via shallow fusion [26].”
Two-Pass End-to-End Speech Recognition (2019, methodology): “Language model rescoring is commonly been used for multi-pass decoding [14, 15, 16, 17], but more recently has been used with a LAS model to rescore hypotheses from a first-pass conventional model [18].”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “Decoding with the external LSTM-based LM trained in the Stage 3 is also conducted (Kannan et al., 2017).”
Towards Online End-to-end Transformer Automatic Speech Recognition (2019, methodology): “An external word-level LM, which was a single-layer LSTM with 1000 units, was used for rescoring using shallow fusion [33] with a weight of 1.”
End-to-End Speech Recognition: A Survey (2023, methodology): “As we described in Section VI-F, shallow fusion simply combines the E2E and LM scores by a log-linear combination as Score( C | X ) = log P ( C | X ) + γ log P ( C ) where γ is a scaling factor for the LM [255], [256], [257].”
Rnn-Transducer with Stateless Prediction Network (2020, methodology): “Many recent works have investigated effective ways of using text-only data to improve E2E models, including fusion with an external LM [4, 5, 6, 7, 8], knowledge distillation from an LM [9], augmenting paired data with TTS data [10, 11], etc.”
Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin (2018, methodology): “Incorporating a character-based language model (LM) into the attention-based models has shown great effectiveness [18, 19].”