SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

By Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Matthew Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, M. Maritxalar, Rada Mihalcea and 3 more. 2015

650 citations recorded by the source corpus.

In semantic textual similarity (STS), systems rate the degree of semantic equivalence between two text snippets. This year, the participants were challenged with new datasets in English and Spanish. The annotations for both subtasks leveraged crowdsourcing. The English subtask attracted 29 teams with 74 system runs, and the Spanish subtask engaged 7 teams participating with 16 system runs. In addition, this year we ran a pilot task on interpretable STS, where the systems needed to add an explanatory layer, that is, they had to align the chunks in the sentence pair, explicitly annotating the kind of relation and the score of the chunk pair. The train and test data were manually annotated by an expert, and included headline and image sentence pairs from previous years. 7 teams participated with 29 runs.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “We include STS tasks from 2012 (Agirre et al., 2012), 2013 4 (Agirre et al., 2013), 2014 (Agirre et al., 2014), 2015 (Agirre et al., 2015) and 2016 (Agirre et al., 2016).”
Generating Datasets with Pretrained Language Models (2021, methodology): “We evaluate D INO on several English semantic textual similarity datasets: the STS tasks 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (STSb) (Cer et al., 2017), and the SICK-Relatedness dataset (SICK) (Marelli et al., 2014).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “On average, our embeddings outperform previous approaches by 0.2% on STS 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), and by 0.9% on STS-Benchmark (Cer et al., 2017).”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “We use the STS tasks 2012 - 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (Cer et al., 2017), and the SICK-Relatedness dataset (Marelli et al., 2014).”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “…2 regularizer on the compositional parameters W c (not applicable for the word averaging model), the pool of phrases used to obtain negative examples (coupled with mini-batch size B , to reduce the number of tunable hyperparameters), λ w , the regularizer on the word embeddings, and δ , the margin.”
SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation (2016, methodology): “The STS shared task has been held annually since 2012, providing a venue for the evaluation of state-of-the-art algorithms and models (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “We evaluate on 7 STS tasks: STS 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014).”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “For example, SentEval [12] contains a collection of natural language tasks, such as sentiment analysis [48, 56], entailment [10] and semantic textual similarity [4, 5, 2, 1, 3].”
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019, methodology): “Our input data come from the SemEval Semantic Textual Similarity tasks from years 2012 - 2016 (Agirre et al., 2012, 2013, 2014, 2015).”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “We test the word representations on the 20 textual similarity datasets from the 2012-2015 Se-mEval semantic textual similarity (STS) shared tasks (Agirre et al., 2012, 2013, 2014; Agirrea et al., 2015), and the 2012 SemEval Semantic Related task (SICK) (Marelli et al., 2014).”
UMDeep at SemEval-2017 Task 1: End-to-End Shared Weight LSTM Model for Semantic Textual Similarity (2017, background): “The data available for this competition has been updated annually and contains gold-label, human-evaluated scores based on sentence pairs across multiple languages ((Agirre et al., 2012), (Agirre et al., 2013), (Agirre et al., 2014), (Agirre et al., 2015)).”