Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

By Yinfei Yang, Gustavo Hernández Ábrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Matthew Cer, Yun-Hsuan Sung, Brian Strope, R. Kurzweil. 2019

124 citations recorded by the source corpus.

In this paper, we present an approach to learn multilingual sentence embeddings using a bi-directional dual-encoder with additive margin softmax. The embeddings are able to achieve state-of-the-art results on the United Nations (UN) parallel corpus retrieval task. In all the languages tested, the system achieves P@1 of 86% or higher. We use pairs retrieved by our approach to train NMT models that achieve similar performance to models trained on gold pairs. We explore simple document-level embeddings constructed by averaging our sentence embeddings. On the UN document-level retrieval task, document embeddings achieve around 97% on P@1 for all experimented language pairs. Lastly, we evaluate the proposed model on the BUCC mining task. The learned embeddings with raw cosine similarity scores achieve competitive results compared to current state-of-the-art models, and with a second-stage scorer we achieve a new state-of-the-art level on this task.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “Contrastive losses are commonly used for retrieval and representation learning (e.g., Yang et al., 2019; 3https://cloud.google.com/translate/ Chen et al., 2020) and in our case exploits all random instruction-trajectory pairs in a minibatch.”
FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “We add bidirectional loss for retrieving speech given a text query and vice-versa [36].”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “Closest to this approach is the research described in España-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “We train an image-text dual encoder (Figure 5) [11, 5, 40] for cross-modal retrieval [9], using the model and code of Parekh et al.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Closest to this approach is the research described in Espa˜na-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “We consider several neural baseline models, all of which are dual encoders (Gillick et al., 2018; Yang et al., 2019a) that encoding left and right inputs separately, as illustrated in Figure 8.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “These include bilingual dual encoder architectures [71, 201], and shared multilingual encoder-decoder architectures [161].”
Language-agnostic BERT Sentence Embedding (2020, methodology): “Dual encoder models are an effective approach for learning cross-lingual embeddings (Guo et al., 2018; Yang et al., 2019a).”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, background): “3https://github.com/google-research/ bert PAWS-X (Yang et al., 2019b) is a paraphrase detection dataset.”
Multilingual Universal Sentence Encoder for Semantic Retrieval (2019, background): “Table 3 shows precision@1 (P@1) for the proposed models as well as the current state-of-the-art results from Yang et al. (2019), which uses a dual-encoder architecture trained on mined bilingual data.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, methodology): “Recent works focus on dual encoders, for example (Yang et al., 2019a; Reimers and Gurevych, 2019; Yang et al., 2019b; Feng et al., 2020).”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, methodology): “L MMS has three main differences with Yang et al. (2019): (1) a masking term that accounts for the fact that there might be multiple positive choices in the batch for a given input; (2) a varying margin term δ , which is increased during training; (3) a log term that makes MMS more closely resemble…”
Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “Yu et al. (2018); Yang et al. (2019) uses a similar cosine-based loss for multilingual training.”
Large Dual Encoders Are Generalizable Retrievers (2021, methodology): “(2) P We apply this loss function in a bi-directional way (Yang et al., 2019) for matchings of both question-to-document and document-to-question.”
ReQA: An Evaluation for End-to-End Answer Retrieval Models (2019, methodology): “…usually follow a two-step approach to answer a given question: first retrieve relevant articles or blocks, and then scan the returned text to identify the answer using a reading comprehension model (Jurafsky and Martin, 2018; Kratzwald and Feuerriegel, 2018; Yang et al., 2019a; Lee et al., 2019).”