A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition

By Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N. Sainath, Karen Livescu. 2018

171 citations recorded by the source corpus.

Attention-based recurrent neural encoder-decoder models present an elegant solution to the automatic speech recognition problem. This approach folds the acoustic model, pronunciation model, and language model into a single network and requires only a parallel corpus of speech and text for training. However, unlike in conventional approaches that combine separate acoustic and language models, it is not clear how to use additional (unpaired) text. While there has been previous work on methods addressing this problem, a thorough comparison among methods is still lacking. In this paper, we compare a suite of past methods and some of our own proposed methods for using unpaired text data to improve encoder-decoder models. For evaluation, we use the medium-sized Switchboard data set and the large-scale Google voice search and dictation data sets. Our results confirm the benefits of using unpaired text across a range of methods and data sets. Surprisingly, for first-pass decoding, the rather simple approach of shallow fusion performs best across data sets. However, for Google data sets we find that cold fusion has a lower oracle error rate and outperforms other approaches after second-pass rescoring on the Google voice search data set.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “To further enhance end-to-end system’s performance, we train byte-pair encoding (BPE)-level language models and apply them to the system by shallow fusion [35,36].”
Masked Language Model Scoring (2019, methodology): “Although end-to-end models f = PS2S(W |X) predict W directly from X , interpolation with the unconditional g = PLM(W ) remains helpful (Toshniwal et al., 2018).”
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, background): “The output probabilities of the new words can be increased by modifying the decoder to incorporate a biased language model [6].”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “For further comparison, we also report the WERs of our best wordpiece model combined with an LSTM language model [34] by shallow fusion [35, 36](6).”
Language Modeling with Deep Transformers (2019, methodology): “We apply our word-level models to hybrid speech recognition by lattice rescoring [20], and the BPElevel models to end-to-end models by shallow fusion [21, 22].”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, methodology): “Another usage of the external LM is to initialize the lower layer in the decoder network with the pre-trained LM [19, 22].”
A comprehensive analysis on attention models (2019, methodology): “…et al., 2014, Luong et al., 2015, Wu et al., 2016] has been successfully applied to speech recognition [Chan et al., 2015, Chiu et al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b] and is one promising end-to-end approach, i.e. the models output are words,…”
A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition (2019, background): “The most popular of these is “Shallow Fusion” [5, 6, 7, 8, 9], which is simple loglinear interpolation between the scores from the end-to-end model and the separately-trained LM.”
Semi-supervised Training for End-to-end Models via Weak Distillation (2019, methodology): “Overall leveraging text-only data has shown between 3% to 7% relative improvement in WER for ASR [3].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “method is shallow fusion in which the external LM is log-linearly interpolated with the E2E model at inference time [275].”
SpeechBrain: A General-Purpose Speech Toolkit (2021, unknown): “CTC [24] Transducers [25] CTC+Attention [26] Shallow fusion [27] LibriSpeech [28] Common Voice [29] AISHELL [30] TIMIT [31] separation, pyannote [17] and sidekit [18] for speaker diarization, and s3prl [19] for self-supervised speech representations.”
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, methodology): “Shallow fusion [21], is a simple, effective and commonly used technique for external language model integration in sequence to sequence learning for ASR [26].”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “We apply the language model in decoding by shallow fusion [56,57].”
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR (2020, unknown): “Language model fusion approaches such as shallow fusion [6], deep fusion [7], and cold fusion [8, 9] utilize an external LM during beam-search decoding.”
Rnn-Transducer with Stateless Prediction Network (2020, methodology): “Many recent works have investigated effective ways of using text-only data to improve E2E models, including fusion with an external LM [4, 5, 6, 7, 8], knowledge distillation from an LM [9], augmenting paired data with TTS data [10, 11], etc.”