SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

By Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, Roberto Zamparelli. 2014

450 citations recorded by the source corpus.

This paper presents the task on the evaluation of Compositional Distributional Semantics Models on full sentences organized for the first time within SemEval2014. Participation was open to systems based on any approach. Systems were presented with pairs of sentences and were evaluated on their ability to predict human judgments on (i) semantic relatedness and (ii) entailment. The task attracted 21 teams, most of which participated in both subtasks. We received 17 submissions in the relatedness subtask (for a total of 66 runs) and 18 in the entailment subtask (65 runs).

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Layer Normalization (2016, methodology): “We checkpoint both models after every 50,000 iterations and evaluate their performance on ﬁve tasks: semantic-relatedness (SICK) [Marelli et al., 2014], movie review sentiment (MR) [Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity classiﬁcation (SUBJ)…”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “Data Collection In data collection for NLI, different annotator decisions about the coreference between entities and events across the pair can lead to very different assignments of labels (de Marneffe et al., 2008; Marelli et al., 2014a; Bowman et al., 2015).”
A large annotated corpus for learning natural language inference (2015, methodology): “The data for the SemEval 2014 task called Sentences Involving Compositional Knowledge (SICK) is a step up in terms of size, but only to 4,500 training examples, and its partly automatic construction introduced some spurious patterns into the data (Marelli et al. 2014a, § 6).”
Skip-Thought Vectors (2015, methodology): “Our ﬁrst experiment is on the SemEval 2014 Task 1: semantic relatedness SICK dataset [29].”
A SICK cure for the evaluation of compositional distributional semantic models (2014, methodology): “SICK was used in SemEval 2014 Task 1 9 ; details on the shared task results are reported in Marelli et al. (2014).”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, methodology): “In Multilayer LSTM architectures, the hidden state of an LSTM unit in layer ` is used as input to the LSTM unit in layer `+1 in the same time step (Graves et al., 2013; Sutskever et al., 2014; Zaremba and Sutskever, 2014).”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “(2015) and Li et al. (2015a) require significant training time on GPUs, on the order of multiple weeks.”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, methodology): “Semantic relatedness: For the semantic relatedness task, we used the SICK dataset (Marelli et al., 2014), and followed the standard split for the training, development, and test sets.”
Learning to Generate Reviews and Discovering Sentiment (2017, methodology): “While our model performs competitively on Microsoft Research Paraphrase Corpus (Dolan et al., 2004) in Table 3, it performs poorly on the SICK semantic relatedness task (Marelli et al., 2014) in Table 4.”
A Deep Reinforcement Learning Chatbot (2017, methodology): “…movie plot summaries and actor biographies extracted from the IMDB database (Mikolov et al. 2013 SkipThought Vector Models : The system contains a SkipThought Vector model (Kiros et al. 2015) trained on the BookCorpus dataset (Zhu et al. 2015) and on the SemEval 2014 Task 1 (Marelli et al. 2014).”
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (2018, methodology): “ew empirical method to investigate a quality of a large corpus designed for the recognizing textual entailment (RTE) task (Condoravdiet al., 2003; Bos and Markert, 2005; MacCartney and Manning, 2009; Marelli et al., 2014a). The proposed method, which is inspired by a statistical hypothesis test, ass”