Simple Fusion: Return of the Language Model

By Felix Stahlberg, James Cross, Veselin Stoyanov. 2018

78 citations recorded by the source corpus.

Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the training data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architecture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to utilize monolingual data such as backtranslation, shallow fusion, and cold fusion.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Post-norm Geometric Mean Fusion In addition to minimizing KL divergence, which affects the past via ∆ H t , we perform post-norm fusion similarly to Stahlberg et al. (2018).”
Masked Language Model Scoring (2019, methodology): “Incidentally, we have demonstrated conclusive improvements on Transformers via LM rescoring for the ﬁrst time, despite only using N -best lists; the most recent fusion work (Stahlberg et al., 2018) only used LSTM-based models.”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “, 2015), integration of a language model into the decoder (Gulcehre et al., 2017; Stahlberg et al., 2018) all the way to fully unsupervised approaches (Lample et al.”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, methodology): “…semisupervised methods relying on back-translation (Sennrich et al., 2015), integration of a language model into the decoder (Gulcehre et al., 2017; Stahlberg et al., 2018) all the way to fully unsupervised approaches (Lample et al., 2018b; Artetxe et al., 2018), which use monolingual data both…”
Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection (2019, unknown): “We plan to investigate ways to further preserve both sentiment and context information by using cold fusion [33] or simple fusion [34].”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “In shallow fusion, the LM is used to score the candidate words generated by the decoder of the NMT system either during inference time [70, 169] or training time [173].”
Plug-and-Play Conversational Models (2020, methodology): “In addition, we also use the Post-norm Geometric Fusion (Stahlberg et al., 2018; Dathathri et al., 2019)”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, background): “Table 1 shows results for all model, as well as unconditional GPT-2 and the results using Simple Fusion (Stahlberg et al., 2018).”
Neural Machine Translation: A Review (2019, background): “, 2019), or in source sentence order (Stahlberg et al., 2018).”