SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

By Eneko Agirre, Carmen Banea, Daniel Matthew Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, Janyce Wiebe. 2016

Comunicacio presentada al 10th International Workshop on Semantic Evaluation (SemEval-2016), celebrat els dies 16 i 17 de juny de 2016 a San Diego, California.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “We include STS tasks from 2012 (Agirre et al., 2012), 2013 4 (Agirre et al., 2013), 2014 (Agirre et al., 2014), 2015 (Agirre et al., 2015) and 2016 (Agirre et al., 2016).”
Generating Datasets with Pretrained Language Models (2021, methodology): “Unfortunately, obtaining large amounts of high-quality training data can be both difficult and prohibitively expensive (Bowman et al., 2015; Agirre et al., 2016).”
Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “We evaluated our approach for continual sentence representation learning using semantic textual similarity (STS) datasets (Agirre et al., 2012, 2013, 2014, 2015, 2016).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “We evaluate on 7 STS tasks: STS 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014).”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “For example, SentEval [12] contains a collection of natural language tasks, such as sentiment analysis [48, 56], entailment [10] and semantic textual similarity [4, 5, 2, 1, 3].”
Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora (2017, methodology): “was performed according to the guidelines of the SemEval 2016 cross-language sentence similarity task (Agirre et al., 2016).”
Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext (2017, methodology): “4 As development data, we use the 2016 STS tasks (Agirre et al., 2016), where the tuning criterion is the average Pearson’s r over its 5 datasets.”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, methodology): “This task was first introduced in the 2016 edition of SemEval (Agirre et al., 2016).”
Whitening Sentence Representations for Better Semantics and Faster Retrieval (2021, methodology): “7 datasets including STS 2012-2016 tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (Cer et al., 2017) and the SICK-Relatedness dataset (Marelli et al., 2014) are adopted as our benchmarks for evalutation.”
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021, methodology): “On average, ConSERT base joint Method STS12 STS13 STS14 STS15 STS16 STSb SICK-R Avg. achieves a performance gain of 2.88 over the re-implemented SBERT base -NLI, and ConSERT large joint achieves a performance gain of 2.70.”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “Our primary evaluation are the 2012-2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016), where the goal is to accurately predict the degree to which two sentences have the same meaning as measured by human judges.”
Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer (2018, methodology): “Meaning Preservation: Following the annotation scheme developed for the Semantic Textual Similarity (STS) dataset (Agirre et al., 2016), given two sentences i.e. the source style sentence and the target style reference rewrite or the target style model output, workers rate the meaning similarity of…”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, unknown): “Besides, there are more significant empirical advantages on retrieval, STS, pair classification, and re-rank.”
Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data (2020, background): “Many self-supervised pretraining schemes for NLP produce vector representations of sentences in which distance in vector space closely tracks semantic similarity (see e.g. recent results on the semantic textual similarity benchmark; Agirre et al., 2016).”