Reframing Human-AI Collaboration for Generating Free-Text Explanations

By Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark O. Riedl, Yejin Choi. 2021

176 citations recorded by the source corpus.

Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written examples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite the intrinsic subjectivity of acceptability judgments, we demonstrate that acceptability is partially correlated with various fine-grained attributes of explanations. Our approach is able to consistently filter GPT-3-generated explanations deemed acceptable by humans.

Subfields: Artificial Intelligence, Information Systems and Management, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yejin Choi
  3. Percy Liang
  4. Luke S. Zettlemoyer
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Quoc V. Le
  8. Graham Neubig

In Information Systems and Management

  1. Omar Shaikh
  2. Xiang Ren
  3. Mohit Bansal
  4. Yejin Choi
  5. James L. McClelland
  6. Greg Durrett
  7. Quoc V. Le
  8. Ana Marasovic

In Natural Language Processing

  1. Mohit Bansal
  2. Yejin Choi
  3. Greg Durrett
  4. Luke S. Zettlemoyer
  5. Kyunghyun Cho
  6. Noah A. Smith
  7. Hannaneh Hajishirzi
  8. Jianfeng Gao

In Alignment & RLHF

  1. Yu Meng
  2. Yejin Choi
  3. Diyi Yang
  4. Rohan Taori
  5. Denny Zhou

Citations in the corpus