Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models

By Takashi Wada, Tomoharu Iwata. 2018

28 citations recorded by the source corpus.

We propose an unsupervised method to obtain cross-lingual embeddings without any parallel data or pre-trained word embeddings. The proposed model, which we call multilingual neural language models, takes sentences of multiple languages as an input. The proposed model contains bidirectional LSTMs that perform as forward and backward language models, and these networks are shared among all the languages. The other parameters, i.e. word embeddings and linear transformation between hidden states and outputs, are specific to each language. The shared LSTMs can capture the common sentence structure among all languages. Accordingly, word embeddings of each language are mapped into a common latent space, making it possible to measure the similarity of words across multiple languages. We evaluate the quality of the cross-lingual word embeddings on a word alignment task. Our experiments demonstrate that our model can obtain cross-lingual embeddings of much higher quality than existing unsupervised models when only a small amount of monolingual data (i.e. 50k sentences) are available, or the domains of monolingual data are different across languages.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “…including aligning multilingual word embeddings (Mikolov et al., 2013; Chen and Cardie, 2018; Lample et al., 2018b) into universal space, and learning crosslingual models (Wada and Iwata, 2018; Lample and Conneau, 2019; Conneau et al., 2019) to exploit shared representations across languages.”
Cross-lingual Language Model Pretraining (2019, background): “The most similar work to ours is probably the one of Wada and Iwata (2018), where the authors train a LSTM (Hochreiter and Schmidhuber, 1997) language model with sentences from different languages.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Recently these approaches have been extended from single language training to cross-lingual training [34, 15, 4, 19].”
Large-Scale Transfer Learning for Natural Language Generation (2019, background): “Works in zero-shot machine translation used large corpora of monolingual data to improve performances for lowresource languages (Johnson et al., 2017; Wada and Iwata, 2018; Lample and Conneau, 2019).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “For example, Wada and Iwata [239] focused on aligning word representations of each language by mapping word embeddings of each language into a common latent space.”