Importance of Search and Evaluation Strategies in Neural Dialogue Modeling

By Ilia Kulikov, Kyunghyun Cho. 2018

93 citations recorded by the source corpus.

We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, methodology): “Work on neural dialog systems have proposed methods for diverse beam search, using a task-speciﬁc diversity scoring function or constraining beam hypotheses to be sufﬁciently different (Li et al., 2016a; Vijayakumar et al., 2018; Kulikov et al., 2019; Pal et al., 2006).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Likert evaluations are often not strong enough to find statistically significant differences between some models, making it difficult to measure incremental improvements (Kulikov et al., 2019).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “The integer scores however suffer from differing bias and variance per annotator, which researchers have tried to mitigate (Kulikov et al. 2018), but nevertheless due to its lack of sensitivity often yields comparisons that are not statistically signiﬁcant.”
Neural Text Generation with Unlikelihood Training (2019, background): “Several forms of diverse beam search have been explored (Li et al., 2016; Vijayakumar et al., 2018; Kulikov et al., 2018; Holtzman et al., 2018) which can to some degree decrease the level of repetition of a model by selecting candidates unlike previously chosen ones, due to their variety.”
Survey of Hallucination in Natural Language Generation (2022, background): “The PersonaChat datasets [31, 228] have accelerated research into persona consistency [68, 91, 126, 208, 219, 224, 232].”
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, background): “Finally, the long-standing problem of dull and repetitive response generation (Li et al., 2017) has been in the focus of recent work (Kulikov et al., 2018; Holtzman et al., 2019).”
Real or Fake? Learning to Discriminate Machine from Human Generated Text (2019, background): “The approach has been recently re-discovered in the context of dialogue modeling by Kulikov et al. (2018), but here negatives are randomly chosen next utterances from the training dataset.”
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “Specifically, human evaluations using direct assessment have been shown to suffer from annotator bias, high variance and sequence effects where the annotation of one item is influenced by preceding items (Kulikov et al., 2019; Sudoh et al., 2021; Liang et al., 2020; See et al., 2019; Mathur et al., ”
Submix: Practical Private Prediction for Large-Scale Language Models (2022, background): “This type of sampling is known to produce unnatural and incoherent text (Kulikov et al., 2018; Holtzman et al., 2019).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “, 2005), leading to different biases per annotator (Kulikov et al., 2019).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, background): “this and tend to generate outputs with low diversity due to standard likelihood training, approximate decoding objectives, and lack of high quality multireference datasets (Fan et al., 2018; Kulikov et al., 2019; Freitag et al., 2020; Choi et al., 2020).”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “The looping phenomenon in the context of beam search decoding with generative models of language was also observed by [44, 26, 82, 43, 63, 108]; we explore sampling-based alternatives to beam search for the other unit types later in Section 4.”
Do Massively Pretrained Language Models Make Better Storytellers? (2019, background): “For any probabilistic language model, the generated text is strongly affected by the choice of decoding algorithm – this is especially true for open-ended text generation tasks such as storytelling and chitchat dialogue (Kulikov et al., 2018; Holtz-man et al., 2019).”