Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

By Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, Jason Aaron Edward Weston. 2019

212 citations recorded by the source corpus.

Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws.In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jason Aaron Edward Weston
  3. Yejin Choi
  4. Jianfeng Gao
  5. Luke S. Zettlemoyer
  6. Percy Liang
  7. Noah A. Smith
  8. Caiming Xiong

In Natural Language Processing

  1. Jason Aaron Edward Weston
  2. Mohit Bansal
  3. Yejin Choi
  4. Douwe Kiela
  5. Kurt Shuster
  6. Emily Dinan
  7. Y-Lan Boureau
  8. Eric Michael Smith

In Alignment & RLHF

  1. Ethan Perez
  2. Christopher D. Manning
  3. Nat McAleese
  4. Yejin Choi
  5. Douwe Kiela
  6. Prithviraj Ammanabrolu
  7. Francis Song
  8. Geoffrey Irving

Citations in the corpus