SemEval-2018 Task 12: The Argument Reasoning Comprehension Task

By Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein. 2018

37 citations recorded by the source corpus.

A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal was to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of about 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. John Lawrence
2. Chris Reed
3. Timothy Niven
4. Hung-Yu kao
5. Samuel R. Bowman
6. Zhiyong Lu
7. Yejin Choi

In Information Systems

1. Chris Reed
2. John Lawrence
3. Timothy Niven
4. Hung-Yu kao
5. Samuel R. Bowman

In Natural Language Processing

1. Chris Reed
2. John Lawrence
3. Timothy Niven
4. Hung-Yu kao
5. Samuel R. Bowman
6. Yejin Choi

In Evaluation & Benchmarks

1. Chris Reed
2. Timothy Niven
3. Hung-Yu kao
4. John Lawrence

Citations in the corpus

• Probing Neural Network Comprehension of Natural Language Arguments (2019, methodology): “The most successful previous work on ARCT (Choi and Lee, 2018; Zhao et al., 2018; Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2017), and utilized effective NLI models such as ESIM (Chen et al., 2016) and InferS”
• Argument Mining: A Survey (2020, background): “With appropriate context, however, the distinction becomes operationally important and was the driver for the first shard task in argument mining, conducted at SEMEVAL2018 by Habernal et al. (2018).”