An Effective Approach to Unsupervised Machine Translation

By Mikel Artetxe, Gorka Labaka, Eneko Agirre. 2019

159 citations recorded by the source corpus.

While machine translation has traditionally relied on large amounts of parallel corpora, a recent research line has managed to train both Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) systems using monolingual corpora only. In this paper, we identify and address several deficiencies of existing unsupervised SMT approaches by exploiting subword information, developing a theoretically well founded unsupervised tuning method, and incorporating a joint refinement procedure. Moreover, we use our improved SMT system to initialize a dual NMT model, which is further fine-tuned through on-the-fly back-translation. Together, we obtain large improvements over the previous state-of-the-art in unsupervised machine translation. For instance, we get 22.5 BLEU points in English-to-German WMT 2014, 5.5 points more than the previous best unsupervised system, and 0.5 points more than the (supervised) shared task winner back in 2014.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “To train SMT models, the unsupervised tuning (Artetxe et al., 2019) was applied to further enhanced the unsupervised SMT system, which employed a small pseudo generated by target-to-source system to adjust weights of source-to-target system.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “Although semi-supervised techniques have been presented as a solution to the scarcity of parallel data, we note the following concerns with respect to their applicability in the context of LRLs: (1) A LRL translation scenario has been simulated by taking small amounts of parallel data from high-reso”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Backtranslation (Edunov et al., 2018; Sennrich et al., 2016a), originally invented for semi-supervised MT, has been employed as a standard approach for unsupervised MT (Lam-ple et al., 2018b,a; Artetxe et al., 2019, 2018).”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “Further combining these with phrase tables from statistical machine translation leads to impressive results (Lample et al., 2018b; Artetxe et al., 2018a; Ren et al., 2019; Artetxe et al., 2019).”
Bilingual Lexicon Induction through Unsupervised Machine Translation (2019, methodology): “So as to optimize the weights of the resulting model, we use the unsupervised tuning procedure proposed by Artetxe et al. (2019), which combines a cyclic consistency loss and a language modeling loss over a subset of 2,000 sentences from each monolingual corpora.”
Language Models are Unsupervised Multitask Learners (2019, methodology): “This outperforms several unsupervised machine translation baselines from (Artetxe et al., 2017) and (Lample et al., 2017) but is still much worse than the 33.5 BLEU of the current best unsupervised machine translation approach (Artetxe et al., 2019).”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “Backtranslation (Edunov et al., 2018; Sennrich et al., 2016a), originally invented for semi-supervised MT, has been employed as a standard approach for unsupervised MT (Lample et al., 2018b;a; Artetxe et al., 2019; 2018).”
A Multilingual View of Unsupervised Machine Translation (2020, methodology): “To optimize our loss, we leverage the EM algorithm, giving a rigorous justification for the stop-gradient operation that is usually applied in the UNMT and style transfer literature (Lample & Conneau, 2019; Artetxe et al., 2019; He et al., 2020).”