SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

By Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Matthew Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, Janyce Wiebe. 2014

610 citations recorded by the source corpus.

In Semantic Textual Similarity, systems rate the degree of semantic equivalence between two text snippets. This year, the participants were challenged with new data sets for English, as well as the introduction of Spanish, as a new language in which to assess semantic similarity. For the English subtask, we exposed the systems to a diversity of testing scenarios, by preparing additional OntoNotesWordNet sense mappings and news headlines, as well as introducing new genres, including image descriptions, DEFT discussion forums, DEFT newswire, and tweet-newswire headline mappings. For Spanish, since, to our knowledge, this is the first time that official evaluations are conducted, we used well-formed text, by featuring sentences extracted from encyclopedic content and newswire. The annotations for both tasks leveraged crowdsourcing. The Spanish subtask engaged 9 teams participating with 22 system runs, and the English subtask attracted 15 teams with 38 system runs.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “The STS14 datasets (Agirre et al., 2014) consist of pairs of sentences annotated by humans with similarity scores.”
Learned in Translation: Contextualized Word Vectors (2017, background): “Hill et al. [2016] study the effect of transferring from a variety of source domains to the semantic similarity tasks in Agirre et al. [2014].”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “To encourage and support research in this area, the STS shared task has been held annually since 2012, providing a venue for evaluation of state-ofthe-art algorithms and models (Agirre et al., 2012, 2013, 2014, 2015, 2016).”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “The STS 2014 dataset (Agirre et al., 2014) consists of 3,750 pairs and ratings from six linguistic domains.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “We include STS tasks from 2012 (Agirre et al., 2012), 2013 4 (Agirre et al., 2013), 2014 (Agirre et al., 2014), 2015 (Agirre et al., 2015) and 2016 (Agirre et al., 2016).”
Generating Sentences by Editing Prototypes (2017, methodology): “10 545 similarity assessments pairs were collected through Amazon Mechanical Turk following Agirre (2014), with the same scale and prompt.”
Generating Datasets with Pretrained Language Models (2021, methodology): “We evaluate DINO on several English semantic textual similarity datasets: the STS tasks 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (STSb) (Cer et al., 2017), and the SICK-Relatedness dataset (SICK) (Marelli et al., 2014).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “On average, our embeddings outperform previous approaches by 0.2% on STS 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), and by 0.9% on STS-Benchmark (Cer et al., 2017).”
Towards Universal Paraphrastic Sentence Embeddings (2015, neutral): “, 2013), 2014 (Agirre et al., 2014), and 2015 (Agirre et al.”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “The STS task was first introduced by Agirre et al. (2012). Early methods focused on lexical semantics, surface form matching and basic syntactic similarity (Bär et al.”
Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “We evaluated our approach for continual sentence representation learning using semantic textual similarity (STS) datasets (Agirre et al., 2012, 2013, 2014, 2015, 2016).”
SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability (2015, methodology): “(6)For additional information, we refer the reader to (Agirre et al., 2014).”
Charagram: Embedding Words and Sentences via Character n-grams (2016, background): “Further details are provided in the official task descriptions (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015).”
SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation (2016, methodology): “The STS shared task has been held annually since 2012, providing a venue for the evaluation of state-of-the-art algorithms and models (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “We evaluate on 7 STS tasks: STS 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014).”
UNAL-NLP: Combining Soft Cardinality Features for Semantic Textual Similarity, Relatedness and Entailment (2014, methodology): “The SemEval-2014 task 10 (multilingual semantic similarity) (Agirre et al., 2014) is the sequel of the semantic textual similarity (STS) evaluations at SemEval in the past two years (Agirre et al., 2012; Agirre et al., 2013).”
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, neutral): “Further details are provided in the ofﬁcial task descriptions (Agirre et al., 2012, 2013, 2014, 2015).”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “For example, SentEval [12] contains a collection of natural language tasks, such as sentiment analysis [48, 56], entailment [10] and semantic textual similarity [4, 5, 2, 1, 3].”