Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation

By Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, Tian Gao. 2019

113 citations recorded by the source corpus.

Introducing common sense to natural language understanding systems has received increasing research attention. It remains a fundamental question on how to evaluate whether a system has the sense-making capability. Existing benchmarks measure common sense knowledge indirectly or without reasoning. In this paper, we release a benchmark to directly test whether a system can differentiate natural language statements that make sense from those that do not make sense. In addition, a system is asked to identify the most crucial reason why a statement does not make sense. We evaluate models trained over large-scale language modeling tasks as well as human performance, showing that there are different challenges for system sense-making.

Subfields: Artificial Intelligence, Safety Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Probing tasks include probing for factual and commonsense knowledge (Petroni et al., 2019; Wang et al., 2019), linguistic capacities (Ettinger, 2020; Kassner and Schtze, 2019), understanding of rare words (Schick and Schütze, 2020), and ability to perform symbolic reasoning (Talmor et al.”
Scaling Instruction-Finetuned Language Models (2022, neutral): “This degradation may be surprising in light of multiple prior studies ﬁnding that instruction ﬁnetuning improves performance on unseen tasks (Wei et al., 2021; Sanh et al., 2021; Wang et al., 2019a; Min et al., 2022, inter alia ).”
Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “The idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., Petroni et al., 2019; Wang et al., 2019; Talmor et al., 2020; Schick and Schütze, 2020; Ettinger, 2020; Kassner and Schütze, 2020; Sakaguchi et al., 2020).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “…consists of one sentence or 2–3 simple sentences, (iii) the task has a ﬁxed set of possible labels, (iv) the explanation is (Camburu et al., 2018), ECQA (Aggarwal et al., 2021), C OM VE (Wang et al., 2019), SBIC (Sap et al., 2020). human-authored, and (v) the dataset has at least 389 instances.”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “The next two, SemEval and TACRED (Hendrickx et al., 2010; Zhang et al., 2017), are relation extraction tasks with a subset of data points annotated by Wang et al. (2019b).”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “…including image cap-tioning datasets, such as Flickr30k (Young et al., 2014), MSCOCO (Lin et al., 2014), Conceptual Captions (Sharma et al., 2018), as well as video captioning datasets including LSMDC (Rohrbach et al., 2017), ActivityNet (Krishna et al., 2017), and VATEX (Wang et al., 2019b).”
Unsupervised Commonsense Question Answering with Self-Talk (2020, methodology): “Alternatively, LM score is used to rank answer choices in a zero-shot setup (Wang et al., 2019; Sakaguchi et al., 2020).”
Explaining Question Answering Models through Text Generation (2020, methodology): “Unlike end-to-end models where reasoning is internal to the model, the hypothesis is an inspectable intermediate representation that exposes relevant knowledge extracted from the LM.”
Evaluating Commonsense in Pre-trained Language Models (2019, background): “2019), Sense Making with Reasoning (Wang et al. 2019), and Argument Reasoning Comprehension (Habernal et al.”
Explanations for CommonsenseQA: New Dataset and Models (2021, background): “Wang et al. (2019) introduced the task of sense-making where given a pair of natural language statements, the goal is to pick the more sensible statement in the pair.”
Language Generation with Multi-hop Reasoning on Commonsense Knowledge Graph (2020, methodology): “We conduct experiments on three commonsense-aware text generation tasks including story ending generation (Mostafazadeh et al., 2016), abductive natural language generation (Bhagavatula et al., 2020), and explanation generation for sense making (Wang et al., 2019).”
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision (2020, background): “This is typically done by training a model on free-form human explanations (Camburu et al. 2018; Rajani et al. 2019; Wang et al. 2019; Zellers et al. 2019), however, such supervision is not always available, and is costly to obtain.”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, background): “Since then, a variety of datasets have been collected with free-form human generated explanations accompanying each data point (Camburu et al., 2018; Kim et al., 2018; Zellers et al., 2019; Wang et al., 2019; Rajani et al., 2019).”
Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning (2021, background): “ize well on textual commonsense reasoning tasks (Wang et al., 2019), Wang et al.”
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022, background): “It aims to generate an explanation given a counterfactual statement for sense-making (Wang et al., 2019).”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, background): “ComVE (Wang et al., 2019) is a dataset for the task of commonsense validation, where from a pair of sentences, a model needs to choose the sentence that defies commonsense (see Fig.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “These collected sentence annotations can be considered sentencelevel highlights, as they point to the sentences in a document that justify each task label (Wang et al., 2019b).”
SemEval-2020 Task 4: Commonsense Validation and Explanation (2020, background): “A pilot study (Wang et al., 2019) has been performed, showing that there is still a signiﬁcant gap between human and machine performance when no training data is provided, despite that the models have already been pretrained with over 100 million natural language sentences.”