SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

By Wei Xu, Chris Callison-Burch, William B. Dolan. 2015

191 citations recorded by the source corpus.

In this shared task, we present evaluations on two related tasks Paraphrase Identification (PI) and Semantic Textual Similarity (SS) systems for the Twitter data. Given a pair of sentences, participants are asked to produce a binary yes/no judgement or a graded score to measure their semantic equivalence. The task features a newly constructed Twitter Paraphrase Corpus that contains 18,762 sentence pairs. A total of 19 teams participated, submitting 36 runs to the PI task and 26 runs to the SS task. The evaluation shows encouraging results and open challenges for future research. The best systems scored a F1-measure of 0.674 for the PI task and a Pearson correlation of 0.619 for the SS task respectively, comparing to a strong baseline using logistic regression model of 0.589 F1 and 0.511 Pearson; while the best SS systems can often reach >0.80 Pearson on well-formed text. This shared task also provides insights into the relation between the PI and SS tasks and suggests the importance to bringing these two research areas together. We make all the data, baseline systems and evaluation scripts publicly available. 1

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, positive): “We also evaluate on the SemEval 2015 Twitter task (Xu et al., 2015) and the SemEval 2014 SICK Semantic Relatedness task (Marelli et al.”
Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext (2017, methodology): “…of semantic relatedness, then evaluate the embeddings on the SemEval semantic textual similarity (STS) tasks from 2012 to 2015 (Agirre et al., 2012, 2013, 2014, 2015), the SemEval 2015 Twitter task (Xu et al., 2015), and the SemEval 2014 SICK Semantic Relatedness task (Marelli et al., 2014).”
Referential translation machines for predicting semantic similarity (2015, background): “ParSS contains sentences provided by Twitter (Xu et al. 2015).”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, methodology): “There are 761 input sentences in Twitter-Para and each input has 9.41 paraphrase candidates on average.”
Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, background): “Many researchers have developed methods for generating or ﬁnding naturally-occurring sentential para-phrase pairs (Barzilay and McKeown, 2001; Dolan et al., 2004; Dolan and Brockett, 2005; Quirk et al., 2004; Coster and Kauchak, 2011; Xu et al., 2014, 2015; Lan et al., 2017).”
Obfuscating Gender in Social Media Writing (2016, unknown): “It would also help to combine word vector based similarity measures with other existing data-driven paraphrase extraction methods (Ganitkevitch et al., 2013; Xu et al., 2014; Xu et al., 2015).”
MTEB: Massive Text Embedding Benchmark (2022, background): “TwitterSemEval2015 (Xu et al., 2015) Paraphrase-Pairs of Tweets from the SemEval 2015 workshop.”
LCQMC:A Large-scale Chinese Question Matching Corpus (2018, methodology): “The other paraphrase corpora mainly come from SemEval task, e.g. the semantic textual similarity corpus(SRA, (Dzikovska et al., 2013)), semantic textual similarity (Agirre et al., 2013), and Twitter paraphrase corpus (Xu et al., 2015; Lan et al., 2017).”
The BQ Corpus: A Large-scale Domain-specific Chinese Corpus For Sentence Semantic Equivalence Identification (2018, methodology): “…several corpora have been provided in recent years, including the Microsoft Research Paraphrase (MSRP) Corpus (Dolan et al., 2004; Dolan and Brockett, 2005), the Twitter Paraphrase Corpus (PIT-2015 corpus) (Xu et al., 2014, 2015), the Twitter URL corpus (Lan et al., 2017) and the Quora dataset 2.”