A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation

By Joshua Albrecht, Rebecca Hwa. 2007

68 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “[17], Cao and Wang [18] Albrecht and Hwa [2], Chen et al.”
Neural Approaches to Conversational AI (2018, neutral): “On the other hand, the fact that this metric is trainable leads to other potential problems such as overfitting and “gaming of the metric” (Albrecht and Hwa, 2007),12 which might explain why previously proposed machine-learned evaluation metrics (Corston-Oliver et al.”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, background): “[2019], machine-learned metrics lead to potential problems such as overfitting and “gaming of the metric” Albrecht and Hwa [2007]. For example, Sai et al.”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, neutral): “2 Measuring sentence-level consistency Because the sentence-level judgments collected in the manual evaluation are relative judgments rather than absolute judgments, it is not possible for us to measure correlation at the sentencelevel in the same way that previous work has done (Kulesza and Shieber”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “ there should be multiple appropriate responses to address the one-to-many essence in open-domain dialog. 7In discussing the potential pitfalls of machine-learned evaluation metrics, Albrecht and Hwa [1] argued for example that it would be “prudent to defend against the potential of a system gaming ”
Further Meta-Evaluation of Machine Translation (2008, background): “Some previous work has focused on developing automatic metrics which predict human ranking at the sentence-level (Kulesza and Shieber, 2004; Albrecht and Hwa, 2007a; Albrecht and Hwa, 2007b).”
Cross-Language Document Summarization Based on Machine Translation Quality Prediction (2010, background): “Albrecht and Hwa (2007) rely on regression algorithms and refer-ence-based features to measure the quality of sentences.”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Some issues are that they may not generalize as well to data beyond that which they are trained (overfit) and also may be biased and gameable (Wu et al., 2019; Albrecht and Hwa, 2007).”