Automated rationale generation: a technique for explainable AI and its effects on human perceptions

By Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, Mark O. Riedl. 2019

283 citations recorded by the source corpus.

Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “Following Ehsan et al. (2019), we analyse the plausibility of explanations which aims to understand how users would perceive such explanations if they were generated by humans.”
Attention is not not Explanation (2019, background): “any accept the notion of extractive methods such asLei et al.(2016), in which explanations come directly from the input itself (as in attention), as plausible. Works such asMullenbach et al.(2018) andEhsan et al. (2019) use human evaluation to evaluate explanations; the former based on attention sco”
Evaluation of Text Generation: A Survey (2020, background): “Explainable AI refers to AI and machine learning methods that can provide human-understandable justifications for their behaviour (Ehsan et al., 2019).”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “Models that in addition to providing an answer can explain their reasoning may have signiﬁcant beneﬁts pertaining to trust and debuggability (Doshi-Velez and Kim, 2017; Ehsan et al., 2019).”
Expanding Explainability: Towards Social Transparency in AI systems (2021, methodology): “Moreover, a notable area of XAI work focuses on generating human-consumable and domain-specific machine explanations by learning from how humans explain [27, 40], which could be a fruitful area to explore when combined with the availability of 4W information.”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “, 2019), playing arcade games (Ehsan et al., 2019), fact checking (Atanasova et al.”
Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach (2020, methodology): “Table 2: Descriptions for the emergent components underlying the human-factor dimensions of the generated rationales (See [20] for further details).”
On the Diversity and Limits of Human Explanations (2021, background): “However, we note exceptions such as explanations of actions in gaming (Ehsan et al., 2019), where humans may often choose sub-optimal actions (labels).”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “There is an emerging area of XAI research on generating human-consumable explanations with the supervision of human explanations [34, 50, 56], which essentially translates model reasoning into meaningful human explanations applied to the same prediction.”
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations (2020, background): “Furthermore, while visual explanations are widely used [29, 40, 42, 47], there is growing interest in verbal explanations using text [18, 27, 55].”
Human-Centered Artificial Intelligence and Machine Learning (2019, unknown): “Ehsan et al. (2019) show that human-like rationales, despite being true reflections of the internal processes of a black-box intelligent system, promote feelings of trust, rapport, familiarity, and comfort in non-experts operating autonomous systems and robots.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “[34]†† playing arcade games crowd 2000 VCR†† [143] visual commonsense reasoning crowd ∼290K E-SNLI-VE†† [32] visual-textual entailment crowd 11,335 (3)‡ ESPRIT†† [101] reasoning about qualitative physics crowd 2441 (2) VLEP†† [72] future event prediction auto + crowd 28,726 EMU†† [27] reasoning abou”
Learning to Faithfully Rationalize by Construction (2020, methodology): “This condition aligns with coherency (Lei et al., 2016), human-likeness and understandability (Ehsan et al., 2019).”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, background): “This resonates with the concept of rationales [15, 17].”