Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models

By Subhajit Naskar, Pedram Rooshenas, Simeng Sun, Mohit Iyyer, Andrew McCallum. 2020

67 citations recorded by the source corpus.

The discrepancy between maximum likelihood estimation (MLE) and task measures such as BLEU score has been studied before for autoregressive neural machine translation (NMT) and resulted in alternative training algorithms (Ranzato et al., 2016; Norouzi et al., 2016; Shen et al., 2016; Wu et al., 2018). However, MLE training remains the de facto approach for autoregressive NMT because of its computational efficiency and stability. Despite this mismatch between the training objective and task measure, we notice that the samples drawn from an MLE-based trained NMT support the desired distribution – there are samples with much higher BLEU score comparing to the beam decoding output. To benefit from this observation, we train an energy-based model to mimic the behavior of the task measure (i.e., the energy-based model assigns lower energy to samples with higher BLEU score), which is resulted in a re-ranking algorithm based on the samples drawn from NMT: energy-based re-ranking (EBR). We use both marginal energy models (over target sentence) and joint energy models (over both source and target sentences). Our EBR with the joint energy model consistently improves the performance of the Transformer-based NMT: +3.7 BLEU points on IWSLT’14 German-English, +3.37 BELU points on Sinhala-English, +1.4 BLEU points on WMT’16 English-German tasks.

Subfields: Artificial Intelligence, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Iyyer
2. Graham Neubig
3. Luke S. Zettlemoyer
4. Mohit Bansal
5. Myle Ott
6. John Wieting
7. Jianfeng Gao
8. Kalpesh Krishna

In Machine Translation

1. Marc'Aurelio Ranzato
2. John Wieting
3. Mohit Iyyer
4. Kalpesh Krishna
5. Myle Ott
6. Ming Zhou

Citations in the corpus

• RankGen: Improving Text Generation with Large Ranking Models (2022, background): “…(Wang and Ou, 2018), open-ended text generation (Bakhtin et al., 2019; Deng et al., 2020), machine translation (Shen et al., 2004; Lee et al., 2021; Bhattacharyya et al., 2021), constrained generation (Qin et al., 2022; Mireshghallah et al., 2022), and models for specific attributes like style…”
• Residual Energy-Based Models for Text (2021, methodology): “Our promising results on language modeling point to future directions in applying the residual EBM approach to other text generation tasks, such as speech recognition and machine translation (Naskar et al., 2020).”
• A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, methodology): “After that, there is a surge of methods based on deep generative models, such as Variational Autoencoders (VAEs) [48, 125, 138, 142, 149, 154], Generative Adversarial Networks (GANs) [117, 140], and Energy-based Models [8, 25, 135, 166].”