SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

By Eneko Agirre, Daniel Matthew Cer, Mona T. Diab, Aitor Gonzalez-Agirre. 2012

995 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “On the one hand, people have measured performance on intrinsic evaluations, e.g. of human judgments of word or sentence similarity ratings (Agirre et al., 2012; Hill et al., 2016b) or of word associations (Vuli´c et al., 2017).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “wo texts, as semantic similarity is vague notion.107 Most research, in fact, use semantic similarity as a shorthand to refer to a series of tasks known as the Semantic Textual Similarity (STS) tasks [Agirre et al., 2012, Cer et al., 2017]. Thus, semantic similarity is operationally deﬁned by the ann”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, methodology): “To assess how the denotational similarities perform on a more established task and domain, we apply them to the 1500 sentence pairs from the MSR Video Description Corpus (Chen and Dolan, 2011) that were annotated for the SemEval 2012 Semantic Textual Similarity (STS) task (Agirre et al., 2012).”
Generating Datasets with Pretrained Language Models (2021, methodology): “We evaluate D INO on several English semantic textual similarity datasets: the STS tasks 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (STSb) (Cer et al., 2017), and the SICK-Relatedness dataset (SICK) (Marelli et al., 2014).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “On average, our embeddings outperform previous approaches by 0.2% on STS 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), and by 0.9% on STS-Benchmark (Cer et al., 2017).”
A SICK cure for the evaluation of compositional distributional semantic models (2014, methodology): “Following STS annotations, we provide a graded relatedness score to which we add the three class annotation “entailed” vs. “contradiction” vs. “unknown” (neutral) introduced in RTE -4.”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “We use the STS tasks 2012 - 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS benchmark (Cer et al., 2017), and the SICK-Relatedness dataset (Marelli et al., 2014).”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “For more details on these tasks please refer to the relevant publications for the 2012 (Agirre et al., 2012 For tuning, we use two datasets that contain PPDB phrase pairs scored by human annotators on the strength of their paraphrase relationship.”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, neutral): “…systems’ overall level of “understanding”, often with tasks such as semantic similarity and entailment (Wang et al., 2018; Bowman et al., 2015; Agirre et al., 2012; Dagan et al., 2005; Bentivogli et al., 2016), and additional work has been done to design curated versions of these tasks to…”
SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment (2014, methodology): “88 (Agirre et al., 2012).”
SemEval-2014 Task 10: Multilingual Semantic Textual Similarity (2014, methodology): “In 2012 we held the ﬁrst pilot task at SemEval 2012, as part of the *SEM 2012 conference, with great success: 35 teams participated with 88 sys-tem runs (Agirre et al., 2012 For STS 2014 we deﬁned two subtasks: English and Spanish.”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “both models show competitive performance. Reddit+SNLI model outperforms SimBow-primary, which is ranked ﬁrst in the ofﬁcial ranking of the 2017 task. 5 Related Work The STS task was ﬁrst introduced byAgirre et al. (2012). Early methods focused on lexical semantics, surface form matching and basic sy”
Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “We evaluated our approach for continual sentence representation learning using semantic textual similarity (STS) datasets (Agirre et al., 2012, 2013, 2014, 2015, 2016).”
SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability (2015, methodology): “In 2012, we held the first pilot task at SemEval 2012, as part of the *SEM 2012 conference, with great success (Agirre et al., 2012).”
Charagram: Embedding Words and Sentences via Character n-grams (2016, background): “Further details are provided in the official task descriptions (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015).”
Ordinal Common-sense Inference (2016, methodology): “Agirre et al. (2012) piloted a Textual Similarity evaluation which has been reﬁned in subsequent years: systems produce scalar values corresponding to predictions of how similar the meaning is between two provided sentences.”
SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation (2016, methodology): “The STS shared task has been held annually since 2012, providing a venue for the evaluation of state-of-the-art algorithms and models (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “We evaluate on 7 STS tasks: STS 2012–2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014).”
UNAL-NLP: Combining Soft Cardinality Features for Semantic Textual Similarity, Relatedness and Entailment (2014, methodology): “, 2014) is the sequel of the semantic textual similarity (STS) evaluations at SemEval in the past two years (Agirre et al., 2012; Agirre et al., 2013).”