WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge

By Hongming Zhang, Xinran Zhao, Yangqiu Song. 2020

60 citations recorded by the source corpus.

In this paper, we present the first comprehensive categorization of essential commonsense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first provide reasons for making correct decisions and then categorize them into six major knowledge categories. By doing so, we better understand the limitation of existing methods (i.e., what kind of knowledge cannot be effectively represented or inferred with existing methods) and shed some light on the commonsense knowledge that we need to acquire in the future for better commonsense reasoning. Moreover, to investigate whether current WSC models can understand the commonsense or they simply solve the WSC questions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representation models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the performance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/HKUST-KnowComp/WinoWhy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, methodology): “We only report automatic metrics (BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2020)) for NLE quality for ComVE, since WinoGrande has only a small number of test NLE instances (which have been used for grounding in our human evaluation – see Appendix ”
PRover: Proof Generation for Interpretable Reasoning over Rules (2020, background): “…generating natural language explana- tions for multiple NLP tasks, including NLI (Camburu et al., 2018), commonsense reasoning (Rajani et al., 2019; Zhang et al., 2020) and generic text classification tasks (Liu et al., 2019a), our novelty lies in generating compositional explanations consisting…”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, methodology): “Related to the Wino-grad Schema Challenge, WinoWhy (Zhang et al., 2020) contains explanations only for the WSC273 test set (Levesque et al., 2012) and does not follow the structure of our commonsense rule-based explanations, thereby leading to label leakage.”
On the Diversity and Limits of Human Explanations (2021, background): “Reference Task Questions and guidelines Zhang et al. (2020) coreference res- olution Given a context and a pronoun reference relationship, write how you would decide the selected candidate is more likely to be referred than the other candidate using natural language.”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, background): “text explain a prediction, (2) free-form or natural language explanations (Camburu et al., 2018; Rajani et al., 2019; Zhang et al., 2020; Kumar and Talukdar, 2020) that are not constrained to the input, and (3) structured explanations that range from”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “…Jham-tani and Clark (2020), Zhang et al. (2020a) and Zellers et al. (2019a) apply a two-stage methodology to explanation collection, by either extracting explanation candidates automatically (Jham-tani and Clark, 2020) or having crowdworkers write them (Zhang et al., 2020a; Zellers et al., 2019a).”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, methodology): “11 [100] commonsense QA crowd 10,962 ECQA [2] commonsense QA crowd 10,962 SEN-MAKING [124] commonsense validation students + authors 2,021 CHANGEMYVIEW [10] argument persuasiveness crowd 37,718 WINOWHY [144] pronoun coreference resolution crowd 273 (5) SBIC [111] social bias inference crowd 48,923 (”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “…(Lin et al., 2018; Sap et al., 2019a,b; Bisk et al., 2020; Hwang et al., 2020; Forbes et al., 2020), pronoun disambiguation (Sakaguchi et al., 2020; Zhang et al., 2020), abductive commonsense reasoning (Bhagavatula et al., 2019) and general commonsense (Talmor et al., 2019; Huang et al., 2019;…”
Prompting Contrastive Explanations for Commonsense Reasoning Tasks (2021, background): “For Winogrande Schemas, ca0 is constructed using the ambiguous pronoun in c (them in Figure 1).”