Learning from Explanations with Neural Execution Tree

By Ziqi Wang, Yujia Qin, Wenxuan Zhou, Jun Yan, Qinyuan Ye, Leonardo Neves, Zhiyuan Liu, Xiang Ren. 2019

41 citations recorded by the source corpus.

While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data, which restricts their applications in scenarios where data annotation is expensive. Natural language (NL) explanations have been demonstrated very useful additional supervision, which can provide sufficient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles. However, directly applying them for augmenting model learning encounters two challenges: (1) NL explanations are unstructured and inherently compositional, which asks for a modularized model to represent their semantics, (2) NL explanations often have large numbers of linguistic variants, resulting in low recall and limited generalization ability. In this paper, we propose a novel Neural Execution Tree (NExT) framework to augment training data for text classification using NL explanations. After transforming NL explanations into executable logical forms by semantic parsing, NExT generalizes different types of actions specified by the logical forms for labeling data instances, which substantially increases the coverage of each NL explanation. Experiments on two NLP tasks (relation extraction and sentiment analysis) demonstrate its superiority over baseline methods. Its extension to multi-hop question answering achieves performance gain with light annotation effort.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Refining Language Models with Compositional Explanations (2021, methodology): “On the other hand, several works [15, 47, 51] argue that natural language explanations rather than numeric labels as human feedback provide a richer and more efficient source of supervision for training.”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “Lastly, a few approaches learn to use explanations for automatically labeling data for data augmentation purposes (Hancock et al., 2018; Wang et al., 2019b; Awasthi et al., 2020), which is effectively fitting to data from a prior distribution given by the labeling mechanism”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “Compared to recent works [7, 13] that parse explanations to logic forms, our approach does not require task-speciﬁc semantic parsers and matching models, making it task-agnostic and applicable to various natural language understanding tasks with minimal additional efforts.”
LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation (2020, methodology): “However, (Wang* et al., 2020) proposes an improved model training paradigm, which relaxes this strict matching constraint, subsequently improving weak labeling coverage and allowing for a larger pool of unlabeled data to be”
Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment (2022, background): “…for document ranking (Joachims, 2002)—502 recent work has explored explicit feedback such as503 explanations of incorrect responses by chatbots (Li504 et al., 2016; Weston, 2016) and correctness labels 505 in conversational question answering and text clas- 506 sification (Campos et al., 2020).”
On the Diversity and Limits of Human Explanations (2021, background): “…“our ruling” or “summing up” Kotonya and Toni (2020) fact verification automatically scraped from the website, we cannot infer the exact question Wang et al. (2020) sentiment anal- ysis & relation extraction Turkers are prompted with a list of selected predicates (see Appendix) and several…”
Teaching Machine Comprehension with Compositional Explanations (2020, methodology): “Following previous work (Srivastava et al., 2017; Wang et al., 2020), we ﬁrst use a Combinatory Categorial Grammar (CCG) based semantic parser P (Zettlemoyer & Collins, 2012) to transform explanations into structured parses.”