Ordinal Common-sense Inference

By Sheng Zhang, Rachel Rudinger, Kevin Duh, Benjamin Van Durme. 2016

124 citations recorded by the source corpus.

Humans have the capacity to draw common-sense inferences from natural language: various things that are likely but not certain to hold based on established discourse, and are rarely stated explicitly. We propose an evaluation of automated common-sense inference based on an extension of recognizing textual entailment: predicting ordinal human responses on the subjective likelihood of an inference holding in a given context. We describe a framework for extracting common-sense knowledge from corpora, which is then used to construct a dataset for this ordinal entailment task. We train a neural sequence-to-sequence model on this dataset, which we use to score and generate possible inferences. Further, we annotate subsets of previously established datasets via our ordinal annotation protocol in order to then analyze the distinctions between these and what we have constructed.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “and commonsense reasoning (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “…most of the existing MRC datasets, all queries and passages in Re Co RD are automatically mined from news articles, which maximally reduces the human elicitation bias (Gordon and Van Durme, 2013; Misra et al., 2016; Zhang et al., 2017), and the data collection method we propose is cost-efﬁcient.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “…et al. (2018); the small-to-medium-data text-understanding tasks collected in NLI format by Poliak et al. (2018); ordinal common sense inference (Zhang et al., 2017); POS tagging on the Penn Treebank (Marcus et al., 1993); supertag-ging on CCGBank (Hockenmaier and Steedman, 2007); and a variant…”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “JHU Ordinal Commonsense Inference requests a label from 1-5 for the plausibility that one situation entails another (Zhang et al., 2017).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, methodology): “In contrast to traditional NLI which explores hard, logical implications, JOCI instead explores likely inferences from the context.”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “Stud-8 These are “Nobody”, “alone”, “no”, and “empty”. ies in eliciting norming data are prone to repeated responses across subjects (McRae et al., 2005) (see discussion in § 2 of (Zhang et al., 2017)).”
Abductive Commonsense Reasoning (2019, background): “, 2018a) and making inferences about plausible events (Maslan et al., 2015; Zhang et al., 2017).”
“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding (2019, background): “Natural language understanding requires the ability to reason with commonsense knowledge (Schu-bert, 2002; Davis, 2014), and the last few years have seen signiﬁcant amount of work in this direction (e.g., Zhang et al. (2017); Bauer et al. (2018); Tandon et al. (2018)).”
Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation (2019, background): “…tasks that require extra knowledge, such as co-reference resolution (Levesque et al., 2012; Morgenstern and Ortiz, 2015), subsequent event prediction (Roemmele et al., 2011; Zhang et al., 2017; Zellers et al., 2018), or reading comprehension (Mostafazadeh et al., 2016; Ostermann et al., 2018b).”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “Natural Language Inference Tasks For the NLI tasks, we collect common-sense entailment judgments from annotators on a 5-point Likert scale on which 1 denotes ‘deﬁnitely contradiction’ and 5 denotes ‘deﬁnitely entailment’, following Zhang et al. (2017).”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, background): “…following Kiela et al. (2018); the small-to-medium-data textunderstanding tasks collected in NLI format by Poliak et al. (2018); ordinal common sense inference (Zhang et al., 2017); POS tagging on the Penn Treebank (Marcus et al., 1993); and supertagging on CCGBank (Hockenmaier & Steedman, 2007).”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, methodology): “Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
Facts2Story: Controlling Text Generation by Key Facts (2020, background): “While being elusive to deﬁne, it is common to expect sufﬁciently high human agreement on the plausibility of events to follow one another (McRae et al., 2005; Zhang et al., 2017).”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, background): “Zhang et al. (2017) propose a framework to generate hypotheses based on context from general world knowledge or neural sequence-to-sequence methods.”
Efficient Online Scalar Annotation with Bounded Support (2018, background): “A response may correspond to a likelihood judgment, e.g., how likely a predicate is factive (Lee et al., 2015), or that some natural language inference may hold (Zhang et al., 2017).”
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (2018, methodology): “We extend various prior works on challenge NLI datasets (Zhang et al., 2017), and define recasting as leveraging existing datasets to create NLI examples (Glickman, 2006; White et al., 2017).”
Event2Mind: Commonsense Inference on Events, Intents, and Reactions (2018, background): “, entailed statements (Zhang et al., 2017; Blouw and Eliasmith, 2018), explanations of causality (Kang et al.”
Temporal Common Sense Acquisition with Minimal Supervision (2020, background): “rarely say the obvious, in order to communicate more efficiently, but sometimes highlight rarities (Schubert, 2002; Van Durme, 2009; Gordon and Van Durme, 2013; Zhang et al., 2017; Bauer et al., 2018; Tandon et al., 2018).”