ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons

By Margaret Li, Jason Aaron Edward Weston, Stephen Roller. 2019

184 citations recorded by the source corpus.

While dialogue remains an important end-goal of natural language research, the difficulty of evaluation is an oft-quoted reason why it remains troublesome to make real progress towards its solution. Evaluation difficulties are actually two-fold: not only do automatic metrics not correlate well with human judgments, but also human judgments themselves are in fact difficult to measure. The two most used human judgment tests, single-turn pairwise evaluation and multi-turn Likert scores, both have serious flaws as we discuss in this work. We instead provide a novel procedure involving comparing two full dialogues, where a human judge is asked to pay attention to only one speaker within each, and make a pairwise judgment. The questions themselves are optimized to maximize the robustness of judgments across different annotators, resulting in better tests. We also show how these tests work in self-play model chat setups, resulting in faster, cheaper tests. We hope these tests become the de facto standard, and will release open-source code to that end.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “We follow the recent best-practice of performing a pairwise comparative evaluation between two systems [30].”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “Selfchats have been used for evaluating model quality (Li et al., 2016, 2019; Ghandeharioun et al., 2019) demonstrating that they are a useful proxy for human-rated quality metrics.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “…“human-likeness” or “engagingness”, either by automatically comparing with a human ground-truth reference, e.g., by using similarity metrics such as BLEURT (Sellam et al., 2020) or BERTscore (Zhang et al., 2020a), or by asking humans to evaluate this manually (Deriu et al., 2020; Li et al., 2019).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “We use the ACUTE eval (Li et al., 2019) method of evaluating conversational quality, as used for BlenderBot (Roller et al., 2020) and elsewhere.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “19). It furthermore removes any noise produced across multiple turns, wherein a system would be required to ingest its own responses in the conversation history, rather than some produced by a human (Li, Weston, and Roller, 2019). Another common evaluation framework is multi-turn Likert evaluation (”
Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, neutral): “We note that since reporting our results, the authors of Li et al. (2019) have conducted a human evaluation study on ConvAI2, in which our Poly-encoder architecture outperformed all other models compared against, both generative and retrieval based, including the winners of the competition.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Pairwise Comparison We conducted a further ACUTE-Eval (Li et al., 2019) human evaluation where crowdworkers compared chat logs pairwise and gave reasons why one is preferred over the other (see Appendix Table 9 for further details).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “We use the dialogue evaluation system Acute-Eval (Li et al., 2019) to ask evaluators to compare pairs of conversations from models and decide which model generates (i) more biased dialogues and (ii) more engaging dialogues.”
Neural Text Generation with Unlikelihood Training (2019, methodology): “Thus we adopt a 16-layer Transformer architecture, with 8 attention heads, an embedding dimension of 1024, and a fully-connected dimension of 4096 ; the architecture is based on (Baevski and Auli, 2019) but with standard embedding and softmax layers.”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “Results from self-chat experiments highly correlate with those of human-chat experiments, for most, but not all systems (Li et al., 2019b).”
Bot-Adversarial Dialogue for Safe Conversational Agents (2021, methodology): “We use the question corresponding to evaluating en-gaginess from Li et al. (2019), which is phrased as “Who would you prefer to talk to for a long conversation?”.”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “A separate set of crowdworkers are then used to compare models pairwise following the ACUTE-Eval procedure of (Li et al., 2019b), where they are asked to choose which is “the more engaging response” for Image Chat (1500 trials) and “Who would you prefer to talk to for a long conversation?” for WoW…”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, methodology): “Human Evaluation Finally, we perform a human evaluation using the ACUTE-EVAL framework (Li et al., 2019), comparing the MLE base-line to UL for various α .”
SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures (2021, methodology): “To measure engagingness, we use the question ‘Which response would you prefer, in the context of a long conversation?’ reworded from ’Who would you prefer to talk to for a long conversation?’ in (Li et al., 2019).”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, neutral): “We follow the setup and use the analysis questions proposed in the Acute-Eval dialogue evaluation system (Li et al., 2019).”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, methodology): “We then ask crowdworkers to rate these two axes of performance – Knowledge-able and Engagingness – following Li et al. (2019).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Given that human evaluation results can differ significantly with small alterations to instructions or the provided UI (Xu et al., 2020; Li et al., 2019; Novikova et al., 2018), which makes them hard to replicate and compare (Howcroft et al., 2020), we recommend using the provided tooling as a way…”
Multi-Modal Open-Domain Dialogue (2020, methodology): “We then perform ACUTE-Evals (Li et al., 2019b) on the collected conversations of MMB Style and BlenderBot in order for crowdsourced raters to directly compare conversations from different models in an A/B setting.”