The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

By Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein. 2017

164 citations recorded by the source corpus.

Reasoning is a crucial part of natural language argumentation. To comprehend an argument, one must analyze its warrant, which explains why its claim follows from its premises. As arguments are highly contextualized, warrants are usually presupposed and left implicit. Thus, the comprehension does not only require language understanding and logic skills, but also depends on common sense. In this paper we develop a methodology for reconstructing warrants systematically. We operationalize it in a scalable crowdsourcing process, resulting in a freely licensed dataset with warrants for 2k authentic arguments from news comments. On this basis, we present a new challenging task, the argument reasoning comprehension task. Given an argument with a claim and a premise, the goal is to choose the correct implicit warrant from two options. Both warrants are plausible and lexically close, but lead to contradicting claims. A solution to this task will define a substantial step towards automatic warrant reconstruction. However, experiments with several neural attention and language models reveal that current approaches do not suffice.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Probing Neural Network Comprehension of Natural Language Arguments (2019, methodology): “The most successful previous work on ARCT (Choi and Lee, 2018; Zhao et al., 2018; Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2017), and utilized effective NLI models such as ESIM (Chen et al., 2016) and InferS”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “Niven and Kao (2019) ﬁnd that BERT’s unreason-ably good performance on the English Argument Reasoning Comprehension Task (Habernal et al., 2018) falls back to chance if the dataset is modiﬁed by adding adversarial examples that just negate one piece of the original, thus mirroring the distribution…”
Evaluating Commonsense in Pre-trained Language Models (2019, methodology): “2019), and Argument Reasoning Comprehension (Habernal et al. 2018).”
NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension (2018, background): “ARCT isolates the reasoning step by not requiring warrants to be discovered.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “Similarly, the authors of [104] ﬁnd that BERT models trained on comprehension datasets (e.g. ARCT [54]) exploit the presence of negation operators, and removing such cues drops the model to random chance accuracy.”
BLCU_NLP at SemEval-2018 Task 12: An Ensemble Model for Argument Reasoning Based on Hierarchical Attention (2018, methodology): “The difﬁculty of the task is the warrants are plausible and lexically close ∗ *The corresponding author but lead to contradicting claims (Habernal et al., 2018).”
SemEval-2018 Task 12: The Argument Reasoning Comprehension Task (2018, methodology): “Further computational baseline approaches, such as a language model, are evaluated in (Habernal et al., 2018), but we did not consider them within the ofﬁcial competition.”
When Choosing Plausible Alternatives, Clever Hans can be Clever (2019, methodology): “gument Reasoning Comprehension Task (Habernal et al., 2018).”
GIST at SemEval-2018 Task 12: A network transferring inference knowledge to Argument Reasoning Comprehension task (2018, background): “The Argument Reasoning Comprehension is a very important task because “argument comprehension requires not only language understanding and logic skills, but it also heavily depends on common sense” , as mentioned by Habernal et al. (2018).”
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2021, background): “Argument reasoning comprehension (Habernal et al., 2018) is a task closer to NLI which also concerns passage-level logical reasoning.”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “Beyond RTE, the choices in the COPA (Roemmele et al., 2011) dataset, where the task is to ﬁnish a given passage (similar to ROC Stories), and ARCT (Habernal et al., 2018) where the task is to select whether a statement warrants a claim, contain words that correlate with the expected prediction (Kavu”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “…6 diﬀerent benchmarks are Sense Making [Wang et al., 2019], the Winograd Schema Challenge [Levesque et al., 2012], SWAG [Zellers et al., 2018], HellaSwag [Zellers et al., 2019], Sense Making with Reasoning [Wang et al., 2019], and the Argument Reasoning Comprehension Task [Habernal et al., 2018].”
The debate over understanding in AI’s large language models (2022, background): “As an example, consider one such benchmark, the Argument Reasoning Comprehension Task [36].”
Zero-Shot Stance Detection: A Dataset and Model Using Generalized Topic Representations (2020, methodology): “In a similar vein, Habernal et al. (2018) use comments from news articles and manually construct position statements.”
Frame- and Entity-Based Knowledge for Common-Sense Argumentative Reasoning (2018, methodology): “Albeit positive the results do not seem be a strong support for the hypothesis of (Habernal et al., 2018) about external knowledge being beneﬁcial for the deﬁned task, as we observe only moderate improvements.”
Comparing Test Sets with Item Response Theory (2021, background): “ARCT The Argument Reasoning Comprehension Task (Habernal et al., 2018) is a multiplechoice question answering dataset.”
Critical Thinking for Language Models (2020, methodology): “The Argument Reasoning Comprehension (ARC) dataset (Habernal et al., 2018)”