Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

By Shikhar Sharma, Layla El Asri, Hannes Schulz, Jeremie Zumer. 2017

232 citations recorded by the source corpus.

Automated metrics such as BLEU are widely used in the machine translation literature. They have also been used recently in the dialogue community for evaluating dialogue response generation. However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting. Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity. It is thus reasonable to think that these automated metrics would correlate well with human judgment in the task-oriented setting where the generation task consists of translating dialogue acts into a sentence. We conduct an empirical study to confirm whether this is the case. Our findings indicate that these automated metrics have stronger correlation with human judgments in the task-oriented setting compared to what has been observed in the non task-oriented setting. We also observe that these metrics correlate even better for datasets which provide multiple ground truth reference sentences. In addition, we show that some of the currently available corpora for task-oriented language generation can be solved with simple models and advocate for more challenging datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “BUTLER::B RAIN is a novel text-based game agent that generates high-level text actions in a token-by-token fashion akin to Natural Language Generation (NLG) approaches for dialogue (Sharma et al., 2017) and summarization (Gehrmann et al., 2018).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Furthermore, we demonstrate that our LongForm models – instruction-tuned on LongForm-C– outperform prior instruction-tuned baselines such as FLAN-T5 and Alpaca by a big margin on a wide variety of long text generation tasks.”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “Sharma et al. (2018) explore using average-pooling and max-pooling on word embeddings to construct sentence-level representation, which is used to compute cosine similarity between the reference and candidate sentences.”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “All metrics where computed using the nlg-eval toolkit (Sharma et al., 2017).”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, methodology): “We conduct automatic evaluation with metrics: BLEU 1, BLEU 2, BLEU 3, BLEU 4 (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and ROUGE-L (Lin, 2004), and use evaluation package released by (Sharma et al., 2017) to compute them.”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “…that n-gram based evaluation metrics such as BLEU and METEOR do not correlate well with human performance on non-task-oriented dialogue datasets, recently (Sharma et al., 2017) have shown that these metrics can show comparatively stronger correlation with human assessment on task-oriented datasets.”
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, positive): “The implementation provided by Sharma et al. (2017) is used.”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, background): “2015), embedding average cosine similarity ( AVER ; Sharma et al., 2017), Vector extrema cosine similarity ( VECTOR ; Forgues et al., 2014), Greedy Match ( GREEDY ; Rus and Lintean, 2012).”
Frustratingly Hard Evidence Retrieval for QA Over Books (2020, methodology): “6We used an open-source evaluation library (Sharma et al., 2017): https://github.com/Maluuba/nlg-eval.”
A Recurrent BERT-based Model for Question Generation (2019, methodology): “We use the evaluation package released by (Sharma et al., 2017).”
Evaluating Style Transfer for Text (2019, unknown): “, 2014), or word embedding averages (Sharma et al., 2017).”
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021, methodology): “Automated evaluation metrics consists of automatic BLEU scores, METEOR scores, ROUGE_L scores, and CiDER scores between generated and ground truth sentences (Sharma et al., 2017).”
Fast Domain Adaptation for Goal-Oriented Dialogue Using a Hybrid Generative-Retrieval Transformer (2020, methodology): “As set in DSTC8, we evaluate our model on MetaLWOz on word-overlap metrics such as BLEU-1–3, CIDEr, METEOR, ROUGE-L using the NLGEval package [18].”