Training Classifiers with Natural Language Explanations

By Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, Christopher R'e. 2018

160 citations recorded by the source corpus.

Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100 faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Several papers have leveraged semantic parsing to convert natural language explanations into features (Srivastava et al., 2017) or additional training labels (Hancock et al., 2018).”
Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2022), speed up training (Hancock et al., 2018), mitigate bias (Dua et al.”
ALICE: Active Learning with Contrastive Natural Language Explanations (2020, methodology): “” Extracted Knowledge: Pay attention to [Bill] when classifying Ring-billed Gull v.s. California Gulls We choose a simple rule-based semantic parser for simplicity, following Hancock et al. (2018).”
ExpBERT: Representation Engineering with Natural Language Explanations (2020, methodology): “We exclude these results on TACRED as it was not studied in Hancock et al. (2018), so we did not have a corresponding semantic parser and set of programmatic explanations.”
Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “Unlike this work, we study a setting where we have no language at test time and test tasks are unseen, so language from training can no longer be used as additional class information [cf. 16] or weak supervision for labeling additional in-domain data [cf. 15].”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, positive): “, 2019) and relation extraction have observed similar results (Bunescu and Mooney, 2007; Hancock et al., 2018; Ratner et al., 2017).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “On the other hand, a standard approach to generating free-text explanations is to use human-written explanations (Hendricks et al., 2016; Kim et al., 2018; Hancock et al., 2018; Camburu et al., 2018; Ehsan et al., 2018; Liu et al., 2019; Wu and Mooney, 2019; Narang et al., 2020; Tang et al., 2020; M”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “While the no-ops we used consisted only of a few words, in some contexts full natural language explanations may provide usefully relevant information [39].”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “…2015; Zhang et al., 2016; Srivastava et al., 2017; Andreas et al., 2018; Liang et al., 2020), to automatically label data for data augmentation (Hancock et al., 2018; Wang et al., 2019a; Awasthi et al., 2020), as additional supervision (Narang et al., 2020; Hase et al., 2020; Pruthi et al.,…”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “This is the most common approach to free-text rationalization (Hendricks et al., 2016; Kim et al., 2018; Hancock et al., 2018; Camburu et al., 2018; Ehsan et al., 2018; Liu et al., 2019a; Wu and Mooney, 2019; Narang et al., 2020; Do et al., 2020; Tang et al., 2020), but little is understood about…”
Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019, neutral): “Human-generated natural language explanations for clas-siﬁcation data have been used in the past to train a semantic parser that in turn generates more noisy labeled data which can used to train a classiﬁer (Hancock et al., 2018).”
Explaining Answers with Entailment Trees (2021, background): “, 2019), or the syntactic pattern used to locate the answer (Ye et al., 2020; Hancock et al., 2018).”
Towards Explainable NLP: A Generative Explanation Framework for Text Classification (2018, methodology): “Deep learning methods have produced state-of-the-art results in many natural language processing (NLP) tasks (Vaswani et al., 2017; Yin et al., 2018; Peters et al., 2018; Wang et al., 2018; Han-cock et al., 2018; Ma et al., 2018).”
Effective Distant Supervision for Temporal Relation Extraction (2020, background): “A related concept is the idea of labeling functions (Ratner et al., 2016; Hancock et al., 2018), which are used to automatically construct training data for new domains.”
Evaluating and Characterizing Human Rationales (2020, unknown): “This result is a hopeful sign for the topic area of learning-from-explanation, which seeks to use explanations as additional training supervision for models (Hancock et al., 2018; Zaidan and Eisner, 2008).”
The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “With respect to weak supervision, we build on the paradigm introduced in the Snorkel system [27, 29], and extended thereafter [3, 12, 32, 40, 41]; for a more comprehensive review of prior and related weak super-vision work, such as distant supervision [1, 15, 23] and crowdsourcing [4, 8, 9, 18,…”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “…semantic parsing, learning through interaction (Li et al., 2017a; Hancock et al., 2019; Li et al., 2017b, inter alia) and learning from natural language supervision (Srivastava et al., 2017; Co-Reyes et al., 2019; Srivastava et al., 2018; Hancock et al., 2018; Ling and Fidler, 2017, inter alia).”