The AMU-UEDIN Submission to the WMT16 News Translation Task: Attention-based NMT Models as Feature Functions in Phrase-based SMT

By Marcin Junczys-Dowmunt, Tomasz Dwojak, Rico Sennrich. 2016

60 citations recorded by the source corpus.

This paper describes the AMU-UEDIN submissions to the WMT 2016 shared task on news translation. We explore methods of decode-time integration of attention-based neural translation models with phrase-based statistical machine translation. Efficient batch-algorithms for GPU-querying are proposed and implemented. For English-Russian, our system stays behind the state-of-the-art pure neural models in terms of BLEU. Among restricted systems, manual evaluation places it in the first cluster tied with the pure neural model. For the Russian-English task, our submission achieves the top BLEU result, outperforming the best pure neural system by 1.1 BLEU points and our own phrase-based baseline by 1.6 BLEU. After manual evaluation, this system is the best restricted system in its own cluster. In follow-up experiments we improve results by additional 0.8 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Machine Translation

Citations in the corpus

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “We re-frain from using checkpoint averaging (exponential moving averages of parameters) (Junczys-Dowmunt et al., 2016) or checkpoint ensembles (Jean et al., 2015; Chen et al., 2017) to focus on evaluating the performance of individual models.”
Edinburgh Neural Machine Translation Systems for WMT 16 (2016, neutral): “Our models are also used in QT21-HimL-SysComb (Peter et al., 2016), ranked 1–2 for EN → RO, and in AMU-UEDIN (Junczys-Dowmunt et al., 2016), ranked 2–3 for EN RU, and 1–2 for RU EN.”
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “This problem is less severe in some tasks (e.g., image classification and text classification), but it has a significant impact on NMT as machine translation is a challenging task containing numerous translation rules, which are easily forgotten during the training process.”
A Challenge Set Approach to Evaluating Machine Translation (2017, methodology): “Since then, controlled comparisons have used BLEU to show that NMT out-performs strong PBMT systems on 30 translation directions from the United Nations Parallel Corpus (Junczys-Dowmunt et al., 2016a), and on the IWSLT English-Arabic tasks (Durrani et al., 2016).”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “3 Effect of Parameter Smoothing Parameter smoothing (checkpoint averaging (Junczys-Dowmunt, Dwojak, and Sennrich 2016)) is a technique which aims to smooth point estimates of the learned parameters by averaging n steps from the training run and using it for inference.”
Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals (2020, methodology): “Checkpoint averaging has been ﬁ rst used in NMT by Junczys-Dowmunt et al. 34 , who report that averaging four checkpoints is “ not much worse than the actual ensemble ” of the same four checkpoints and it is better than ensembles of two checkpoints.”
Code-Switching for Enhancing NMT with Pre-Specified Translation (2019, methodology): “We use label smoothing (Junczys-Dowmunt et al., 2016) with a confidence score of 0.”
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (2017, methodology): “This was the case of the best submission in terms of BLEU for RU→EN (Junczys-Dowmunt et al., 2016).”
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions (2016, background): “An alternative found to work by [18] is checkpoint averaging, where a single model is produced by element-wise averaging of all corresponding model parameters, producing a single model.”
The University of Cambridge’s Machine Translation Systems for WMT18 (2018, methodology): “Furthermore, we also report gains from MBR-based combination with a phrase-based SMT system.”
Neural Machine Translation: A Review (2019, methodology): “Checkpoint averaging [28, 159] is a technique which is often discussed in conjunction with ensembling [160].”
Neural Machine Translation (2016, unknown): “, 2015] incorporating NMT as a feature function into PBSMT [Junczys-Dowmunt et al., 2016] → results depend on relative performance of PBSMT and NMT log-linear combination of different neural models left-to-right and right-to-left [Liu et al.”
Bi-Directional Neural Machine Translation with Synthetic Parallel Data (2018, methodology): “For DE↔EN experiments, we train a single model and average the parameters of the best four checkpoints for decoding (Junczys-Dowmunt et al., 2016).”