Pushing the Limits of Translation Quality Estimation

By André F. T. Martins, Marcin Junczys-Dowmunt, Fabio Kepler, Ramón Fernandez Astudillo, Chris Hokamp, Roman Grundkiewicz. 2017

76 citations recorded by the source corpus.

Translation quality estimation is a task of growing importance in NLP, due to its potential to reduce post-editing human effort in disruptive ways. However, this potential is currently limited by the relatively low accuracy of existing systems. In this paper, we achieve remarkable improvements by exploiting synergies between the related tasks of word-level quality estimation and automatic post-editing. First, we stack a new, carefully engineered, neural model into a rich feature-based word-level quality estimation system. Then, we use the output of an automatic post-editing system as an extra feature, obtaining striking results on WMT16: a word-level FMULT1 score of 57.47% (an absolute gain of +7.95% over the current state of the art), and a Pearson correlation score of 65.56% for sentence-level HTER prediction (an absolute gain of +13.36%).

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “State-of-the-art models for sQE such as (Martins et al., 2017) and (Kim et al.”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “The “full-stacked-src-mt” system incorporates the output of an APE system, converted to OK/BAD tags, as an additional feature, similar to their work in (Martins et al., 2017a).”
SUM-QE: a BERT-based Summary Quality Estimation Model (2019, methodology): “The scores of their model correlate well with Pyramid and Responsiveness, but text quality is only addressed indirectly.2 Quality Estimation is well established in MT (Callison-Burch et al., 2012; Bojar et al., 2016, 2017; Martins et al., 2017; Specia et al., 2018).”
Ensembling Factored Neural Machine Translation Models for Automatic Post-Editing and Quality Estimation (2017, methodology): “This approach builds upon the approach presented in Martins et al. (2017), by incorporating features which have proven effective for Word Level QE as "factors" in the input to Neural Machine Translation (NMT) systems.”