deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

By Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, William B. Dolan. 2015

159 citations recorded by the source corpus.

We introduce Discriminative BLEU (∆BLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs. Reference strings are scored for quality by human raters on a scale of [−1, +1] to weight multi-reference BLEU. In tasks involving generation of conversational responses, ∆BLEU correlates reasonably with human judgments and outperforms sentence-level and IBM BLEU in terms of both Spearman’s ρ and Kendall’s τ .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Knowledge-Grounded Neural Conversation Model (2017, background): “In terms of BLEU scores, MT ASK -RF improvements is not signiﬁcant, but it generates the highest chine Translation (Graham et al., 2015), as B LEU and related metrics were originally designed as corpus-level metrics.”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “Accurate evaluation of a non-goal-driven dialogue system is an open problem (Galley et al. 2015; Pietquin and Hastie 2013; Schatzmann, Georgila, and Young 2005).”
BERTScore: Evaluating Text Generation with BERT (2019, background): “∆BLEU (Galley et al., 2015) modifies multi-reference BLEU by including human annotated negative reference sentences.”
Why We Need New Evaluation Metrics for NLG (2017, background): “This problem could possibly be solved by weighting multiple references according to their quality, as suggested by (Galley et al., 2015), or following”
Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, methodology): “We use BLEU2 and RougeL (F1) scores to capture word overlap between the generated inference and the references, which captures quality of generation (Galley et al., 2015; Hashimoto et al., 2019).”
Neural Approaches to Conversational AI (2018, neutral): “deltaBLEU (Galley et al., 2015) is an extension of BLEU that exploits numerical ratings associated with conversational responses.”
A Persona-Based Neural Conversation Model (2016, methodology): “ et al., 2016) we used BLEU (Papineni et al., 2002) for parameter tuning and evaluation. BLEU has been shown to correlate well with human judgment on the response generation task, as demonstrated in (Galley et al., 2015). Besides BLEU scores, we also report perplexity as an indicator of model capabi”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, background): “Screenshots and further details of the experiments are in Appendix 6.4.”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “This is because Galley et al. measure corpus-level correlation, i.e. correlation averaged across different subsets (of size 100) of the data, and pre-filter for high-quality reference responses. studies (Liu et al., 2016; Galley et al., 2015).”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “More recently, researchers have turned to analyzing the collected dialogues produced after they are finished (Galley et al., 2015; Pietquin and Hastie, 2013; Shawar and Atwell, 2007a; Schatzmann et al., 2005).”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “Generating text is a core part of many NLP tasks such as image captioning (Lin et al., 2014), open-domain dialogue (Sordoni et al., 2015), story generation (Roemmele, 2016), and summarization (Nallapati et al., 2016).”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, background): “deltaBLEU (Galley et al. 2015) is an extension to BLEU that exploits numerical ratings associated with conversational responses.”
Analyzing Uncertainty in Neural Machine Translation (2018, neutral): “In another context, uncertainty has been considered for the design of better evaluation metrics (Dreyer & Marcu, 2012; Galley et al., 2015), in order not to penalize a model for producing a valid translation which is different from the provided reference.”
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, background): “Also, as observed by Galley et al. (2015), we can see that word-overlap based metrics correlate better with human evaluation when multiple references are provided, as in the Restaurants dataset.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[32] showed that the correlation of string-based metrics (BLEU and deltaBLEU) significantly increases with the units of measurement bigger than a sentence.”
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, methodology): “Multiple reference responses can be obtained with retrieval models (Galley et al., 2015; Sordoni et al., 2015) or through data collection (Gupta et al., 2019).”