A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition

By Erik McDermott, Hasim Sak, Ehsan Variani. 2019

121 citations recorded by the source corpus.

This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “Compared with JATD, a more closely related area of MUTE is the subtraction of internal LMs for end-to-end ASR models [30, 31].”
On the limit of English conversational speech recognition (2021, methodology): “External language model fusion through probability ratio: as proposed in [37], to decode with an end-to-end sequence posterior p(w 1 |x1 ) and external language model p(w 1 ), the language model learned by the E2E model p(w 1 ) has to be compensated.”
Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition (2020, methodology): “In addition to decoding only with a RNN-T model, we also integrate an external LM using the recently proposed density ratio fusion technique(4) [43].”
Have Best of Both Worlds: Two-Pass Hybrid and E2E Cascading Framework for Speech Recognition (2021, background): “Despite a lot of efforts to tackle the issues above in E2E ASR systems [8, 10, 11, 12, 13, 14, 15, 16, 17], it is fair to say that hybrid modelling is still more effective and practical to solve all the above problems due to its modualized structure and technique maturity.”
End-to-End Speech Recognition: A Survey (2023, methodology): “However, due to the implicit prior of E2E models, i.e. the internal language model, combination with separate language models is not straightforward and requires corresponding internal language model estimation and compensation approaches, e.g. [47], [128], [303], [304], [310].”