SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

By Daniel Matthew Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, Lucia Specia. 2017

2,135 citations recorded by the source corpus.

Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “STS-B The Semantic Textual Similarity Benchmark is a collection of sentence pairs drawn from news headlines and other sources (Cer et al., 2017).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “2013) Three tasks involve detecting semantic similarity: Microsoft Research Paraphrase Corpus (MRPC, (Dolan and Brockett, 2005)), Quora Question Pairs1 (QQP), and Semantic Textual Similarity Benchmark (STS-B; Cer et al. 2017).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, neutral): “…9https://www.tensorflow.org/datasets • Paraphrasing/sentence similarity (MRPC [Dolan and Brockett, 2005], STS-B [Cer et al., 2017], QQP [Iyer et al., 2017]) • Natural language inference (MNLI [Williams et al., 2017], QNLI [Rajpurkar et al., 2016], RTE…”
Adversarial Training for Large Neural Language Models (2020, background): “…answering (Rajpurkar et al., 2016), linguistic acceptability (Warstadt et al., 2018), sentiment analysis (Socher et al., 2013), text similarity (Cer et al., 2017), paraphrase detection (Dolan and Brockett, 2005), and natural language inference (NLI) (Da-gan et al., 2006; Bar-Haim et al., 2006;…”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “We also utilize these datasets for semantic similarity, reading comprehension, and textual entailment: • STS-B requires predicting the semantic simi- larity between pairs of sentences (Cer et al., 2017).”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “…(SST; Socher et al., 2013), Microsoft Research Paraphrase Corpus (MRPC; Dolan & Brockett, 2005), Semantic Textual Similarity Benchmark (STS; Cer et al., 2017), Quora Question Pairs (QQP; Iyer et al., 2017), Multi-Genre NLI (MNLI; Williams et al., 2018), Question NLI (QNLI; Rajpurkar et…”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “, 2013), 3 sentence similarity tasks: MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), QQP (Chen et al.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “…in GLUE can be found in Wang et al. (2019a) and in Warstadt et al. (2019, CoLA), Socher et al. (2013, SST-2), Dolan and Brockett (2005, MRPC), Cer et al. (2017, STS-B), and Williams et al. (2018, MNLI), and Rajpurkar et al. (2016, the original data source for QNLI). remains substantial scope…”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “The Semantic Textual Similarity Benchmark is a collection of sentence pairs collected from multiple data resources including news head-lines, video, and image captions, and NLI data (Cer et al., 2017).”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “, 2013); semantic similarity with the MSR Paraphrase Corpus (MRPC; Dolan and Brockett, 2005) and STSBenchmark (STS; Cer et al., 2017); and textual entailment with a subset of the RTE challenge corpora (Dagan et al.”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “…answering (Rajpurkar et al., 2016), linguistic acceptability (Warstadt et al., 2018), sentiment analysis (Socher et al., 2013), text similarity (Cer et al., 2017), paraphrase detection (Dolan and Brockett, 2005), and natural language inference (NLI) (Williams et al., 2018; Levesque et al.,…”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “The Semantic Textual Similarity Benchmark (STS-B; Cer et al., 2017) requires predicting a similarity score between 1 and 5 for a sentence pair; we report the Spearman correlation coefﬁcient (scc).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “sentiment classiﬁcation with SST (Socher et al., 2013); semantic similarity with the MSR Paraphrase Corpus (MRPC; Dolan & Brockett, 2005), the Quora Question Pairs2 (QQP), and STS-Benchmark (STS; Cer et al., 2017); and textual entailment with the Multi-Genre NLI Corpus (MNLI Williams et al., 201”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “Entailment and semantic relatedness We also include the SICK dataset (Marelli et al., 2014) for entailment (SICK-E), and semantic relatedness datasets including SICK-R and the STS Benchmark dataset (Cer et al., 2017).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “In addition to the Reuters corpus, Cer et al. (2017) propose sentence-level multilingual training and evaluation datasets for semantic textual similarity in four languages.”