Learning Semantic Textual Similarity from Conversations

By Yinfei Yang, Steve Yuan, Daniel Matthew Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, R. Kurzweil. 2018

167 citations recorded by the source corpus.

We present a novel approach to learn representations for sentence-level semantic similarity using conversational data. Our method trains an unsupervised model to predict conversational responses. The resulting sentence embeddings perform well on the Semantic Textual Similarity (STS) Benchmark and SemEval 2017’s Community Question Answering (CQA) question similarity subtask. Performance is further improved by introducing multitask training, combining conversational response prediction and natural language inference. Extensive experiments show the proposed model achieves the best performance among all neural models on the STS Benchmark and is competitive with the state-of-the-art feature engineered and mixed systems for both tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “…2017); POS tagging on the Penn Treebank (Marcus et al., 1993); supertagging on CCGBank (Hockenmaier and Steedman, 2007); and a variant objective on our Reddit data, inspired by Yang et al. (2018), where the model is trained to select which of two candidate replies to a given comment is correct.”
ERNIE: Enhanced Representation through Knowledge Integration (2019, methodology): “Sentence encoder (Yang et al., 2018) based on response prediction ben-eﬁts from query-response pair data drawn from Reddit conversation.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Pushshift.io Reddit We use a variant of Reddit discussions, which has also been used in several existing studies (see e.g. Yang et al. (2018); Mazaré et al. (2018); Shuster et al. (2020)).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “Bias is Amplified in Generation Existing Transformer generative dialogue models (Serban et al., 2016; Yang et al., 2018; Urbanek et al., 2019) are trained to take as input the dialogue context and generate the next utterance.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “In addition it uses Pushshift.io Reddit, a variant of Reddit discussions, which has also been used in several existing studies (see e.g., Yang et al. (2018); Mazaré et al. (2018); Shuster et al. (2020)).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “pushshift.io Reddit We use a variant of Reddit discussions, which has also been used in several existing studies, see e.g. Yang et al. (2018); Mazaré et al. (2018); Keskar et al. (2019); Shuster et al. (2019).”
Training Millions of Personalized Dialogue Agents (2018, methodology): “Based solely on attention mechanisms, it exhibited state-of-the-art performance on next utterance retrieval tasks in dialogues (Yang et al., 2018).”
Universal Sentence Encoder for English (2018, methodology): “On the STS Benchmark, we compare with InferSent and the state-of-the-art neural STS systems CNN (HCTI) (Shao, 2017) and gConv (Yang et al., 2018).”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “…the collection, and we thus measure in experiments its ability to help performance on other tasks. pushshift.io Reddit We use a variant of Reddit discussions (text-only), which has also been used in several existing studies, see e.g. Yang et al. (2018); Mazar´e et al. (2018); Keskar et al. (2019).”
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling (2018, background): “A number of variants of sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014; Kalchbrenner and Blunsom, 2013) have been proposed for dialogue modeling in recent years, including hierarchical models (Serban et al., 2016) and transformers (Mazaré et al., 2018; Yang et al., 2018).”
Hash Layers For Large Sparse Models (2021, methodology): “Pushshift.io Reddit We use a variant of Reddit discussions, which has also been used in several existing studies, see e.g. [26, 27, 28, 29].”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, methodology): “Yang et al. (2018) presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset.”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “Reddit These models reconstruct comment threads from reddit.com using a dataset of about 18M comment–response pairs collected from 2008-2011 by Yang et al. (2018).”
Information Leakage in Embedding Models (2020, neutral): “A popular architecture for unsupervised sentence embedding is the dual-encoder model proposed in many prior works [5, 7, 26, 39, 40, 58, 75].”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, methodology): “Our preliminary models make use of the random sampling strategy that has been proven successful in prior work (Henderson et al., 2017; Yang et al., 2018).”
Self-supervised Learning for Large-scale Item Recommendations (2020, background): “More recently, a line of research [27, 44] considers applying two-tower DNNs on retrieval problems, which is also known as dual-encoder [18, 43], where item embeddings are constructed by a MLP from ID and other categorical metadata features.”
Situated Dialogue Learning through Procedural Environment Generation (2021, methodology): “This dataset has been used in several existing dialoguebased studies and has been shown to result in more natural conversations (Yang et al., 2018; Mazaré et al., 2018).”