Cold Fusion: Training Seq2Seq Models Together with Language Models

By Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, Adam Coates. 2017

311 citations recorded by the source corpus.

Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks which involve generating natural language sentences such as machine translation, image captioning and speech recognition. Performance has further been improved by leveraging unlabeled data, often in the form of a language model. In this work, we present the Cold Fusion method, which leverages a pre-trained language model during training, and show its effectiveness on the speech recognition task. We show that Seq2Seq models with Cold Fusion are able to better utilize language information enjoying i) faster convergence and better generalization, and ii) almost complete transfer to a new domain while using less than 10% of the labeled training data.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…settings, including research on back-translation (Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and…”
Hierarchical Neural Story Generation (2018, methodology): “Sriram et al. (2017) propose training the seq2seq model given the ﬁxed language model then learning a gate to ﬁlter the information from the language model.”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “Still other results show that fusing an LM with an AM during training can improve performance (Sriram et al., 2017; Chorowski & Jaitly, 2016; Wu et al., 2016).”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “The most common method for leveraging text-only data is to train a language model (LM) and integrate it into the recognition process using shallow [17], cold [18], or deep fusion [19].”
Masked Language Model Scoring (2019, background): “One explanation comes from cold and simple fusion (Sriram et al., 2018; Stahlberg et al., 2018), which further improve on shallow fusion (Section 3.”
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, methodology): “This includes techniques such as biasing [7, 8, 9], shallow fusion [10], deep fusion [10] and cold fusion [11].”
Optimal Completion Distillation for Sequence Learning (2018, neutral): “8 Cold Fusion (Sriram et al., 2018) 3.”
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model (2017, methodology): “Previous works have focused on n-gram LMs when applying shallow fusion [8, 2] or RNN LMs for deep or cold fusion [7, 11].”
Improving Abstraction in Text Summarization (2018, neutral): “(8) in a fashion similar to Sriram et al. (2017).”
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, background): “Related work on how to best integrate language models into end-to-end ASR includes deep and shallow fusion [21], cold fusion [22], or transfer fusion [23].”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, background): “Prior work focuses on building LM with monolingual text data, and then integrate LM or transfer knowledge from it into the decoder [13, 15, 16].”
Semi-supervised Training for End-to-end Models via Weak Distillation (2019, methodology): “Cold fusion [8] has been shown to be an effective strategy for Voice Search.”
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition (2019, background): “Cold fusion and component fusion utilize a pre-trained recurrent neural network language model (RNNLM) and gating mechanism to improve ASR performances [7, 8].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “There are several LM fusion methods, such as shallow fusion [76], deep fusion [76], and cold fusion [263].”
Hybrid Autoregressive Transducer (HAT) (2020, background): “There has been extensive efforts to develop decoding algorithms that can use external LMs, so-called fusion methods [27, 28, 29, 30, 31].”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, methodology): “Recently, several methods to leverage an external LM during training of S2S models are proposed: deep fusion [20] and cold fusion [21].”
Structured Fusion Networks for Dialog (2019, background): “The output of the NLG, in the form of logits at a decoding time-step, is combined with the hidden state of the decoder via cold-fusion (Sriram et al., 2017).”
A Fully Differentiable Beam Search Decoder (2019, background): “Because our work is end-to-end, it is also related to seq2seq learning (Sutskever et al., 2014; Bahdanau et al., 2014; Chan et al., 2017; Wiseman and Rush, 2016; Gehring et al., 2017), and in particular training with existing/external language models (Sriram et al., 2018).”
Can Unconditional Language Models Recover Arbitrary Sentences? (2019, methodology): “Pretrained Recurrent Language Models Pretrained or separately trained language models have largely been used in two contexts: as a feature extractor for downstream tasks and as a scoring function for a task-speciﬁc decoder (Gulcehre et al., 2015; Li et al., 2016; Sriram et al., 2018).”