Using “Annotator Rationales” to Improve Machine Learning for Text Categorization

By Omar Zaidan, Jason Eisner, Christine D. Piatko. 2007

381 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “As examples, it has been shown that natural language intermediate steps can improve performance (Zaidan et al., 2007; Yao et al., 2021; Hase and Bansal, 2022; Gu et al., 2022), improve robustness (Chen et al., 2022), speed up training (Hancock et al., 2018), mitigate bias (Dua et al., 2020), and…”
Shaping Visual Representations with Language for Few-Shot Classification (2019, background): “In this framework, learning with attributes and other domain-specific rationales has been tackled extensively (Zaidan et al., 2007; Donahue and Grauman, 2011; Tokmakov et al., 2019); language less”
Large Language Models Can Self-Improve (2022, neutral): “Majority Voting by answer ···”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Similar to our work, Zaidan et al. (2007) studied the impact of providing explicit supervision via rationales , rather than generating them, for varying fractions of training set in text classiﬁcation.”
Rationale-Augmented Ensembles in Language Models (2022, methodology): “Previous work has also shown that “rationales” can be useful for supervised learning in natural language tasks when added in the training data (Zaidan et al., 2007; Ling et al., 2017; Cobbe et al., 2021; Zelikman et al., 2022), but it remains unclear whether such rationales can be reliably useful…”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “Saliency can also be computed with respect to intermediate values, rather than input features (Ghaeini et al., 2018).7 7Generally, many of the visualization methods are adapted from the vision domain, where they have been extremely popular; see Zhang and Zhu (2018) for a survey.”
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, background): “Some ask annotators to highlight rationales, spans of text indicative of the label [28, 29, 24].”
Rationalizing Neural Predictions (2016, neutral): “Finally, we contrast our work with rationale-based classification (Zaidan et al., 2007; Marshall et al., 2015; Zhang et al., 2016) which seek to improve prediction by relying on richer annotations in the form of human-provided rationales.”
Yin and Yang: Balancing and Answering Binary Visual Questions (2015, positive): “This signal can be used as additional supervision for training models such as [41, 11, 31, 6] that can leverage explanations provided by the annotator in addition to labels.”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “Predating deep learning, (Zaidan et al., 2007) consider the use of “annotator rationales” in sentiment analysis to train support vector machines.”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, methodology): “exists work on incorporating human input into decision boundaries in the form of annotator rationales [Zaidan et al., 2007; Donahue and Grauman, 2011; Zhang et al., 2016].”
Modeling Annotators: A Generative Approach to Learning from Annotator Rationales (2008, methodology): “We also use A0 to compare against the “masking SVM” method and SVM baseline of Zaidan et al. (2007).”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, methodology): “Work on learning from rationales marked by annotators for text classification dates back over a decade (Zaidan et al., 2007).”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, methodology): “Our work is different from approaches that aim to improve clas-siﬁcation using rationales as an additional input (Zaidan et al., 2007; Zaidan and Eisner, 2008; Zhang et al., 2016).”
HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection (2020, background): “Zaidan, Eisner, and Piatko (2007) introduced the concept of using rationales , in which human annotators would high-light a span of text that could support their labeling decision.”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “For the task of sentiment analysis on movie reviews, Zaidan et al. (2007) collected “rationales” where people highlighted portions of the movie reviews that would encourage (or discourage) readers to watch (or avoid) the movie.”