Multilingual Neural Machine Translation with Task-Specific Attention

By Graeme W. Blackwood, Miguel Ballesteros, Tomás Ward. 2018

80 citations recorded by the source corpus.

Multilingual machine translation addresses the task of translating between multiple source and target languages. We propose task-specific attention models, a simple but effective technique for improving the quality of sequence-to-sequence neural multilingual translation. Our approach seeks to retain as much of the parameter sharing generalization of NMT models as possible, while still allowing for language-specific specialization of the attention model to a particular language-pair or task. Our experiments on four languages of the Europarl corpus show that using a target-specific model of attention provides consistent gains in translation quality for all possible translation directions, compared to a model in which all parameters are shared. We observe improved translation quality even in the (extreme) low-resource zero-shot translation directions for which the model never saw explicitly paired parallel data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…sharing small network modules, for example the attention parameters (Firat et al., 2016a), and sharing everything except the attention parameters (Ha et al., 2016b; Blackwood et al., 2018), to sharing all parameters in the model across all language pairs (Johnson et al., 2017; Ha et al., 2016c).”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, methodology): “While prior work have studied the effect of data (Arivazhagan et al., 2019; Wang et al., 2020a), architecture (Blackwood et al., 2018; Sachan & Neubig, 2018; V ´ azquez et al., 2019; Escolano et al., 2020) and scale (Huang et al., 2019b; Lepikhin et al., 2020) on multilingual models, their…”
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2021, methodology): “For learning prompts, we do not apply ad-729 ditional preprocessing on translation tasks except730 Romanian-English translation task, where we use a731 script 8 to remove diacritics in the Romanian side.732 Because the mT5 tokenizer automatically uses Uni-733 code NFKC normalization, which results i”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “, 2019), making architectures dedicated to LS modeling a necessity (Blackwood et al., 2018; Sachan & Neubig, 2018; Zhang et al., 2020b).”
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, methodology): “Previous work has accomplished this in different ways, including: • pre-pending a special target-language token to the input (Wu et al., 2016) • using an additional embedding vector for the target language (Lample and Conneau, 2019) • using unique decoders for each target language (Luong et al., 201”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “The second variant (“JT-S-MT + Dedicated Attention”) (Blackwood et al., 2018) introduces dedicated decoder modules for different tasks.”
A Survey of Multilingual Neural Machine Translation (2019, background): “Blackwood et al. (2018) explored target language, source language and pair specific attention parameters.”
Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021, methodology): “+TS (Blackwood et al., 2018) This method assigns language-specific attention modules to each language pair.”
The Missing Ingredient in Zero-Shot Neural Machine Translation (2019, background): “In contrast, Blackwood et al. (2018) propose sharing all parameters but the attention mechanism.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Some strategies include scheduling (cycle through the bilingual language pairs) [57], using mini-batches that consist of diferent languages [91, 160] or using mini-batches that contain data from the same target [19].”
Three Strategies to Improve One-to-Many Multilingual Translation (2018, methodology): “Owing to excellent translation performance and ease of use, many researchers (Blackwood et al., 2018; Lakew et al., 2018) have conduct translation of multiple languages based on the framework of Johnson et al. (2017) and Ha et al.”
Multilingual Translation from Denoising Pre-Training (2021, background): “Subsequently, work focused on the the model capacity bottleneck, leading to exploration of various parameter sharing strategies (Blackwood et al., 2018; Platanios et al., 2018; Sachan and Neubig, 2018; Lu et al., 2018).”
Using natural language prompts for machine translation (2022, unknown): “…using language embeddings that we add directly to the intermediate representations of the decoder (Fi-rat et al., 2016) or subword embeddings (Wang et al., 2018; Conneau et al., 2019), task-speciﬁc attention (Blackwood et al., 2018), language tags on the decoder side (Liu et al., 2020), etc.”