A Bilingual Generative Transformer for Semantic Sentence Embedding
By John Wieting, Graham Neubig, Taylor Berg-Kirkpatrick. 2019
34 citations recorded by the source corpus.
Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Edouard Grave
- Graham Neubig
- Mohit Bansal
- Furu Wei
- Danqi Chen
- Alexis Conneau
- Jason Aaron Edward Weston
In Natural Language Processing
- Luke S. Zettlemoyer
- Edouard Grave
- Graham Neubig
- Mohit Bansal
- Furu Wei
- Danqi Chen
- Jason Aaron Edward Weston
- Vishrav Chaudhary
In Representation & Self-Supervised Learning
- Alexis Conneau
- Michael Auli
- Danqi Chen
- Vishrav Chaudhary
- Tianyu Gao
- Xingcheng Yao
- Edouard Grave
- Jingfei Du
Citations in the corpus
- Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “…and fine-grained sentiment analysis datasets (SST-2 and SST-5), on product classification (CR) from (Hu and Liu, 2004 (Koehn, 2005) parallel data from English to French, Italian and Indonesian - language pairs that were shown to provide good paraphrastic sentence embeddings (Wieting et al., 2019a).”
- SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “Furthermore, Wieting and Gimpel (2018); Wieting et al. (2020) demonstrate that bilingual and back-translation corpora provide useful supervision for learning semantic similarity.”