Softmax Tempering for Training Neural Machine Translation Models
By Raj Dabre, Atsushi Fujita. 2020
13 citations recorded by the source corpus.
Neural machine translation (NMT) models are typically trained using a softmax cross-entropy loss where the softmax distribution is compared against smoothed gold labels. In low-resource scenarios, NMT models tend to over-fit because the softmax distribution quickly approaches the gold label distribution. To address this issue, we propose to divide the logits by a temperature coefficient, prior to applying softmax, during training. In our experiments on 11 language pairs in the Asian Language Treebank dataset and the WMT 2019 English-to-German translation task, we observed significant improvements in translation quality by up to 3.9 BLEU points. Furthermore, softmax tempering makes the greedy search to be as good as beam search decoding in terms of translation quality, enabling 1.5 to 3.5 times speed-up. We also study the impact of softmax tempering on multilingual NMT and recurrently stacked NMT, both of which aim to reduce the NMT model size by parameter sharing thereby verifying the utility of temperature in developing compact NMT models. Finally, an analysis of softmax entropies and gradients reveal the impact of our method on the internal behavior of NMT models.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Quoc V. Le
- Ruslan Salakhutdinov
- Jianfeng Gao
- Josh Tenenbaum
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Andrea Vedaldi
- Ming-Hsuan Yang
In Natural Language Processing
- Josh Tenenbaum
- Xinyun Chen
- Mohit Bansal
- Yoshua Bengio
- Pushmeet Kohli
- Quoc V. Le
- Percy Liang
- Chuang Gan
Citations in the corpus
- Competition-level code generation with AlphaCode (2022, methodology): “Fine-tune the model on our dataset of competitive programming data, using GOLD (Pang and He, 2020) with tempering (Dabre and Fujita, 2020) as the training objective.”
- Planning with Large Language Models for Code Generation (2023, methodology): “Transformer-based pipelines like AlphaCode follow the tradition of natural language processing and use sampling methods (Fan et al., 2018; Dabre & Fujita, 2020) during the generation process.”