A SICK cure for the evaluation of compositional distributional semantic models

By Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli. 2014

963 citations recorded by the source corpus.

Shared and internationally recognized benchmarks are fundamental for the development of any computational system. We aim to help the research community working on compositional distributional semantic models (CDSMs) by providing SICK (Sentences Involving Compositional Knowldedge), a large size English benchmark tailored for them. SICK consists of about 10,000 English sentence pairs that include many examples of the lexical, syntactic and semantic phenomena that CDSMs are expected to account for, but do not require dealing with other aspects of existing sentential data sets (idiomatic multiword expressions, named entities, telegraphic language) that are not within the scope of CDSMs. By means of crowdsourcing techniques, each pair was annotated for two crucial semantic tasks: relatedness in meaning (with a 5-point rating scale as gold score) and entailment relation between the two elements (with three possible gold labels: entailment, contradiction, and neutral). The SICK data set was used in SemEval-2014 Task 1, and it freely available for research purposes.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “Entailment NLI There has been a long history of NLI benchmarks focusing on linguistic entailment (Cooper et al., 1996; Dagan et al., 2006; Marelli et al., 2014; Bowman et al., 2015; Lai et al., 2017; Williams et al., 2018).”
An efficient framework for learning sentence representations (2018, neutral): “The semantic relatedness task on the SICK dataset (Marelli et al., 2014) involves predicting relatedness scores for a given pair of sentences that correlate well with human judgements.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “entailment strongly cues for semantic relatedness (Marelli et al., 2014), we construct our own sentence pairings to deter gold entailment labels from informing evaluation set STS scores.”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “The SICK dataset (Marelli et al., 2014) consists of 10,000 pairs of sentences and relatedness judgements.”
A large annotated corpus for learning natural language inference (2015, methodology): “Existing resources suffer from a subtler issue that impacts even projects using only humanprovided annotations: indeterminacies of event and entity coreference lead to insurmountable indeterminacy concerning the correct semantic label (de Marneffe et al. 2008 §4.3; Marelli et al. 2014b).”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “how an agent should perform the task (Marelli et al., 2014; Marvin and Linzen, 2018; Warstadt et al., 2019; Ettinger, 2020).”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
Annotation Artifacts in Natural Language Inference Data (2018, background): “Lai and Hockenmaier (2014) demonstrated that lexical features such as the presence of negation, word overlap, and hypernym relations are highly predictive of entailment classes in the SICK dataset (Marelli et al., 2014).”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “STS We employ the Semantic Textual Similarity Benchmark (STS-B; Cer et al., 2017) and SICK-R (Marelli et al., 2014).”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “Entailment and semantic relatedness We also include the SICK dataset (Marelli et al., 2014) for entailment (SICK-E), and semantic relatedness datasets including SICK-R and the STS Benchmark dataset (Cer et al., 2017).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “…which are then passed through feedfor-ward networks that are trained for the following classiﬁcation tasks: (i) Semantic Textual Similarity (STS) (Marelli et al., 2014), (ii) Question Type Classiﬁcation (TREC) (Voorhees and Dang, 2003), (iii) Subjectivity Classiﬁcation (SUBJ) (Pang and Lee,…”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…(Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), superglue-cb (de Marneffe et al., 2019), sick (Marelli et al., 2014) , hate_speech18 (de Gibert et al., 2018), ethos-national_origin (Mollas et al., 2020), ethos-race (Mollas et al., 2020),…”
Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019, background): “For example, for the comparatively easier natural langauge inference task with ﬁxed labels, Bowman et al. (2015) show that the accuracy dropped by 25% when training on SNLI and evaluating on SICK-E (Marelli et al., 2014).”
e-SNLI: Natural Language Inference with Natural Language Explanations (2018, background): “7% when training on SNLI and evaluating on SICK-E [20].”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “tributional semantic models handle “challenging phenomena”, Marelli et al. (2014) introduced SICK, which used rules to expand or normalize existing premises to create more difficult examples.”