Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation

By Nils Reimers, Iryna Gurevych. 2020

1,359 citations recorded by the source corpus.

We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training is lower. We demonstrate the effectiveness of our approach for 10 languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, methodology): “Recall that we use a version of the pretrained Distil-RoBERTA language model [15, 60] to generate embeddings of each user utterance.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “We use the pretrained SBERT (Reimers and Gurevych, 2020) for sentence embeddings of the user request input (for tasks that only have structured input, we embed the linearized structured input) and sample ﬁve most similar examples measured by cosine similarity.”
What Makes Good In-Context Examples for GPT-3? (2021, background): “(Reimers and Gurevych, 2019, 2020) have shown that these ﬁne-tuned encoders have achieved great performance on tasks such as sentence clustering, para-phrase mining, and information retrieval.”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, methodology): “Reference-free metrics We consider the following reference-free metrics: • Multilingual Sentence BERT (mSBERT) (Reimers and Gurevych, 2020) is a multilingual version of Sentence BERT trained on parallel data with multilingual knowledge distillation.”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, neutral): “These include localization data (GNOME, KDE4, Ubuntu, Mozilla-I10n), religious text (JW300 (Agić and Vulić, 2019), Bible-eudin (Christodouloupoulos and Steedman, 2015), Tanzil), GlobalVoices, OpenSubtitles (Lison and Tiedemann, 2016), TED2020 (Reimers and Gurevych, 2020), WikiMatrix (Schwenk et al.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “(7) direction to introduce view dependency to 4 For example, Reimers and Gurevych [67, 68] trains a multi-lingual (more than 50+ languages) text encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese.”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “To filter out the generated question-answer pairs, we apply the following strategy: (1) remove duplicates using the sentence-transformer (Reimers & Gurevych, 2020), (2) discard answers that begin with phrases like “does not specify,” “does not mention,” “does not specifically,” “does not depict,”…”
PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, methodology): “The reference audios were selected from the training set with the highest sentence embedding similarity computed using sentence-BERT [19] between the training texts and the target text.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “The TED talk transcripts translation test set was extracted from OPUS 5 based on the corpus released by Reimers and Gurevych (2020).”
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022, background): “(Reimers and Gurevych, 2020; Thakur et al., 2021; Wang et al., 2021a), which use Siamese neural networks (Koch et al.”
Video Instruction Tuning With Synthetic Data (2024, positive): “To filter out the generated question-answer pairs, we apply the following strategy: (1) remove duplicates using the sentence-transformer (Reimers & Gurevych, 2020), (2) discard answers that begin with phrases like “does not specify,” “does not mention,” “does not specifically,” “does not depict,”…”
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022, positive): “We use Sentence-BERT (Reimers and Gurevych, 2020) to embed all persona sentences and perform k-means clustering on the embeddings to obtain 8 clusters.”
Machine Translation into Low-resource Language Varieties (2021, methodology): “We use 310K NN sentences from WikiMatrix, and TED dev/test sets for both varieties (Reimers and Gurevych, 2020).”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “The embedding based approach can be extended to new languages (Reimers and Gurevych, 2020; Heffernan et al., 2022) or the speech modality (Duquenne et al.”
Cross-lingual and Multilingual CLIP (2022, background): “Such as cross-lingual teacher learning of Sentence Embeddings (Reimers and Gurevych, 2020) and multimodal transfer learning of Text-To-Speech (Jiang et al., 2021).”