ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning

By Swarnadeep Saha, Prateek Yadav, Lisa Bauer, Mohit Bansal. 2021

83 citations recorded by the source corpus.

Recent commonsense-reasoning tasks are typically discriminative in nature, where a model answers a multiple-choice question for a certain context. Discriminative tasks are limiting because they fail to adequately evaluate the model’s ability to reason and explain predictions with underlying commonsense knowledge. They also allow such models to use reasoning shortcuts and not be “right for the right reasons”. In this work, we present ExplaGraphs, a new generative and structured commonsense-reasoning task (and an associated dataset) of explanation graph generation for stance prediction. Specifically, given a belief and an argument, a model has to predict if the argument supports or counters the belief and also generate a commonsense-augmented graph that serves as non-trivial, complete, and unambiguous explanation for the predicted stance. We collect explanation graphs through a novel Create-Verify-And-Refine graph collection framework that improves the graph quality (up to 90%) via multiple rounds of verification and refinement. A significant 79% of our graphs contain external commonsense nodes with diverse structures and reasoning depths. Next, we propose a multi-level evaluation framework, consisting of automatic metrics and human evaluation, that check for the structural and semantic correctness of the generated graphs and their degree of match with ground-truth graphs. Finally, we present several structured, commonsense-augmented, and text generation models as strong starting points for this explanation graph generation task, and observe that there is a large gap with human performance, thereby encouraging future work for this new challenging task.

Subfields: Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “These results establish an upperbound on the plausible free-text explanation generation task, which should guide future work that seeks to improve free-text explanations via neural or neuro-symbolic systems that inject other sources of knowledge (Brahman et al., 2021; Majumder et al., 2021; Saha et ”
A Survey of Large Language Models (2023, unknown): “Complex Reasoning Knowledge Reasoning CSQA [247], StrategyQA [248], ARC [291], BoolQ [315], PIQA [295], SIQA [316], HellaSwag [317], WinoGrande [318], OpenBookQA [302], COPA [319], ScienceQA [301], proScript [320], ProPara [321], ExplaGraphs [322], ProofWriter [323], EntailmentBank [324], ProOntoQA ”
Can Language Models Solve Graph Problems in Natural Language? (2023, background): “, 2021] or an argument explanation graph [Saha et al., 2021], Madaan et al.”
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering (2024, unknown): “We have converted the triplet-form provided in Saha et al. [35] into a standard graph format.”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, background): “…2021) has explored different forms of explanations ranging from extractive ratio-nales (Zaidan et al., 2007; DeYoung et al., 2020), semi-structured, and structured explanations (Jansen et al., 2019; Mostafazadeh et al., 2020; Saha et al., 2021) to free-text explanations (Camburu et al., 2018).”
Language Models of Code are Few-Shot Commonsense Learners (2022, methodology): “…a major remaining challenge is how to leverage LLMs for structured commonsense reasoning , including tasks such as generating event graphs (Tandon et al., 2019), reasoning graphs (Madaan et al., 2021a), scripts (Sakaguchi et al., 2021), and argument explanation graphs (Saha et al., 2021).”