Language-agnostic BERT Sentence Embedding

By Fangxiaoyu Feng, Yinfei Yang, Daniel Matthew Cer, Naveen Arivazhagan, Wei Wang. 2020

1,305 citations recorded by the source corpus.

While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by LASER, while still performing competitively on monolingual transfer learning benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “Following [36, 37], cross-modal embeddings are trained using the additive margin softmax loss with in-batch negative sampling.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “1), we explore dual encoder learning from both image-caption and translation pairs at massive scale: 6 billion translation pairs (Feng et al., 2020) and 1.8 billion image-caption pairs (Jia et al., 2021).”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “LaBSE (Feng et al., 2020) learns cross-lingual sentence embeddings by an additive translation ranking loss.”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, methodology): “Embedding (LaBSE) (Feng et al., 2020) was trained similar to mUSE with a dual-encoder transformer architecture based on BERT with 6 Billion translation pairs for 109 languages.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, background): “2 Another recent multilingual sentence embedding is LaBSE (Feng et al., 2020).”
NewsEmbed: Modeling News through Pre-trained Document Representations (2021, methodology): “[23] use a dual encoder with additive margin softmax on bilingual translation pairs.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Similarity search is frequently used to evaluate multilingual text encoders (Artetxe and Schwenk, 2018; Feng et al., 2020; Heffernan et al., 2022).”
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, methodology): “• Quality: To ensure high quality shots we sort our training data using LaBSE (Feng et al., 2020).”
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2021, background): “, 2018); translation pairs (Yang et al., 2020a; Feng et al., 2020); paraphrasing pairs (Wieting et al.”
A Primer on Pretrained Multilingual Language Models (2021, neutral): “contrastive/margin-based objectives (Feng et al., 2020; Ruder et al., 2021) and result in high-quality sentence retrieval.”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, methodology): “We compute these embeddings using LaBSE (Feng et al., 2020) which is a state-of-the-art multilingual sentence embedding model that encodes text from different languages into a shared embedding space.”
Large Dual Encoders Are Generalizable Retrievers (2021, background): “…being quickly adopted by the community and achieving state-of-the-art performance on multiple retrieval tasks, such as passage retrieval (Karpukhin et al., 2020), question answering (Ahmad et al., 2019), conversational question answering (Qu et al., 2020) and bitext retrieval (Feng et al., 2020).”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, neutral): “• LaBSE (Feng et al., 2020) is a dual-encoder framework.”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “4 Due to the possibility of the incursion of errors during automatic translation, we used the Language-Agnostic BERT Sentence Embeddings (LaBSE) (Feng et al., 2020) of the translations and original sentences to compute their similarity and discarded all sentences below a similarity threshold of 0.”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “There, pretrained language models (LMs) such as BERT [12] or RoBERTa [36] are fine-tuned via twin-network architectures on auxiliary tasks such as semantic textual similarity [45, 25], paraphrasing [55], response retrieval [57, 22, 21, 25], or translation ranking [10, 16].”
Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021, methodology): “The intent classiﬁers using mUSE and LaBSE as sentence encoders are conﬁg-ured as follows.”