SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment

By Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, Roberto Zamparelli. 2016

62 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Samuel R. Bowman
2. Yonatan Belinkov
3. Mohit Bansal
4. Douwe Kiela
5. Graham Neubig
6. Hinrich Schütze
7. Iryna Gurevych
8. Jianfeng Gao

In Natural Language Processing

1. Samuel R. Bowman
2. Iryna Gurevych
3. Hinrich Schütze
4. Yonatan Belinkov
5. John Wieting
6. Douwe Kiela
7. Graham Neubig
8. Mohit Bansal

In Evaluation & Benchmarks

1. Samuel R. Bowman
2. Iryna Gurevych
3. Kawin Ethayarajh
4. John Wieting
5. Hinrich Schütze
6. Mohit Bansal
7. Douwe Kiela
8. Nils Reimers

Citations in the corpus

• SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “Semantic inference tasks related to STS include textual entailment (Bentivogli et al., 2016; Bowman et al., 2015; Dagan et al., 2010), semantic relatedness (Bentivogli et al., 2016) and paraphrase detection (Xu et al., 2015; Ganitkevitch et al., 2013; Dolan et al., 2004).”
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “Data Collection In data collection for NLI, different annotator decisions about the coreference between entities and events across the pair can lead to very different assignments of labels (de Marneffe et al., 2008; Marelli et al., 2014a; Bowman et al., 2015).”
• Probing for semantic evidence of composition by means of simple classification tasks (2016, background): “First, certain cues in this dataset allow for strong performance without composition (for example, as Bentivogli et al. (2016) point out, 86.4% of sentence pairs labeled as CON-TRADICTION can be identified simply by detecting the presence of negation; a similar obser-134 vation is made by Lai and…”
• What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, background): “…often with tasks such as semantic similarity and entailment (Wang et al., 2018; Bowman et al., 2015; Agirre et al., 2012; Dagan et al., 2005; Bentivogli et al., 2016), and additional work has been done to design curated versions of these tasks to test for specific linguistic capabilities…”
• Distributional Models of Word Meaning (2018, background): “Thus, the distributional approach to semantic similarity can be projected from the lexical level up to the level of the sentence and text similarity (Bentivogli et al. 2016).”