Jointly Learning to Align and Translate with Transformer Models

By Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias Paulik. 2019

186 citations recorded by the source corpus.

The state of the art in machine translation (MT) is governed by neural approaches, which typically provide superior translation accuracy over statistical approaches. However, on the closely related task of word alignment, traditional statistical word alignment models often remain the go-to solution. In this paper, we present an approach to train a Transformer model to produce both accurate translations and alignments. We extract discrete alignments from the attention probabilities learnt during regular neural machine translation model training and leverage them in a multi-task framework to optimize towards translation and alignment objectives. We demonstrate that our approach produces competitive results compared to GIZA++ trained IBM alignment models without sacrificing translation accuracy and outperforms previous attempts on Transformer model based word alignment. Finally, by incorporating IBM model alignments into our multi-task training, we report significantly better alignment accuracies compared to GIZA++ on three publicly available data sets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “[55] show that hallucinations are visible in attention matrices.”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, methodology): “For the De-En, Fr-En, Ro-En datasets, we follow the experimental setting of previous work (Zenkel et al., 2019; Garg et al., 2019; Zenkel et al., 2020).”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, methodology): “by incorporating alignment objectives (Garg et al., 2019), or improving the representation through external infor-”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, methodology): “Several methods create alignments from attention matrices (Peter et al., 2017; Li et al., 2018; Zenkel et al., 2019) or pursue a multitask approach for alignment and translation (Chen et al., 2016; Garg et al., 2019).”
Towards Enhancing Faithfulness for Neural Machine Translation (2020, background): “Zhang et al. (2019), Feng et al. (2020) and Garg et al. (2019) proposed to introduce word alignment information in Trans-former to improve translation accuracy.”