Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features?

By David Marecek, H. Çelikkanat, Miikka Silfverberg, Vinit Ravishankar, Jörg Tiedemann. 2020

10 citations recorded by the source corpus.

We investigate the effect of training NMT models on multiple target languages. We hypothesize that the integration of multiple languages and the increase of linguistic diversity will lead to a stronger representation of syntactic and semantic features captured by the model. We test our hypothesis on two different NMT architectures: The widely-used Transformer architecture and the Attention Bridge architecture. We train models on Europarl data and quantify the level of syntactic and semantic information discovered by the models using three different methods: SentEval linguistic probing tasks, an analysis of the attention structures regarding the inherent phrase and dependency information and a structural probe on contextualized word representations. Our results show evidence that with growing number of target languages the Attention Bridge model increasingly picks up certain linguistic properties including some syntactic and semantic aspects of the sentence whereas Transformer models are largely unaffected. The latter also applies to phrase structure and syntactic dependencies that do not seem to be developing in sentence representations when increasing the linguistic diversity in training to translate. This is rather surprising and may hint on the relatively little influence of grammatical structure on language understanding.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “, 2018) or if models encompassed linguistic features (Celikkanat et al., 2020).”