The Third PASCAL Recognizing Textual Entailment Challenge

By Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, William B. Dolan. 2007

733 citations recorded by the source corpus.

This paper presents the Third PASCAL Recognising Textual Entailment Challenge (RTE-3), providing an overview of the dataset creating methodology and the submitted systems. In creating this year’s dataset, a number of longer texts were introduced to make the challenge more oriented to realistic scenarios. Ad-ditionally, a pool of resources was offered so that the participants could share common tools. A pilot task was also set up, aimed at differentiating unknown en-tailments from identified contradictions and providing justifications for overall system decisions. 26 participants submitted 44 runs, using different approaches and generally presenting new entailment models and achieving higher scores than in the previous challenges.

Subfields: Artificial Intelligence, Education, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Education

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, neutral): “, 2006), RTE3 (Giampiccolo et al., 2007), and RTE5 (Bentivogli et al.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “, 2016), Recognizing Textual Entailment (RTE) (Dagan et al., 2006; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) and Winograd NLI (WNLI) (Levesque et al.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “RTE is included in GLUE, and we use the same data and format as GLUE: We merge data from RTE1 (Dagan et al., 2006), RTE2 (Bar Haim et al., 2006), RTE3 (Giampiccolo et al., 2007), and RTE5 (Bentivogli et al., 2009).”
Unifying Question Answering and Text Classification via Span Extraction (2019, background): “Finally, we note, perhaps surprisingly, that question answering datasets (SQuAD and TriviaQA) improve performance of some of the classiﬁcation tasks. from the Wiki version of TriviaQA) and RTE (pre-trained from any of the three datasets).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “ Dataset references: aWarstadt et al.(2018) bSocher et al.(2013) cDolan and Brockett(2005) dCer et al.(2017) eIyer et al.(2017) fWilliams et al. (2018 )gconstructed from SQuAD (Rajpurkar et al.,2016 hGiampiccolo et al. 2007 Table 1: Comparison of methods on the GLUE dev set. , , and indicate statist”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “…including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020) and RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) • Reading comprehension tests the ability to answer a question when given a passage that contains the answer.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, neutral): “…et al., 2018) • MRPC (Dolan and Brockett, 2005) • QNLI (Rajpurkar et al., 2016; Wang et al., 2019b) • RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) • SST-2 (Socher et al., 2013) • STS-B (Cer et al., 2017) • WNLI (Levesque et al., 2012; Wang et…”
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories (2016, background): “The ‘right ending’ can be viewed as ‘entailing’ hypothesis in a classic Recognizing Textual Entail-ment (RTE) framework (Giampiccolo et al., 2007), and ‘wrong’ ending can be seen as the ’contradict-ing’ hypothesis.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, background): “0 [32], RACE [33], ReCoRD [34] and SWAG [35]; (2) Natural Language Inference: MNLI [36]; and (3) NER: CoNLL-2003.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “, 2018), RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), WiC (Pilehvar and CamachoCollados, 2018), and WSC (Levesque et al.”
Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “…2013), CoLA (Warstadt et al., 2019), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), MRPC (Dolan and Brockett, 2005), QQP12 and STS-B (Cer et al., 2017), we follow Zhang et al.…”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…Brockett, 2005), glue-wnli (Levesque et al., 2012), climate_fever (Diggel-mann et al., 2020), glue-rte (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), superglue-cb (de Marneffe et al., 2019), sick (Marelli et al., 2014) , hate_speech18 (de Gibert…”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “…reading comprehension (Khashabi et al. 2018; Zhang et al. 2018), natural language inference (Dagan, Glickman, and Magnini 2006; Bar Haim et al. 2006; Giampiccolo et al. 2007; Ben-tivogli et al. 2009; Poliak et al. 2018), word sense disam-biguation (Pilehvar and Camacho-Collados 2019), winograd…”
Rationale-Augmented Ensembles in Language Models (2022, methodology): “…Language Inference : For these tasks, we include ANLI (Nie et al., 2020) with the three subsets (R1, R2, R3), e-SNLI (Camburu et al., 2018), MNLI (matched/mis-matched) (Williams et al., 2018), and RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “Recognizing Textual Entailment [4, 9, 16, 18] aims to predict whether a given premise sentence entails a given hypothesis sentence or not.”