Are BLEU and Meaning Representation in Opposition?

By Ondřej Cífka, Ondrej Bojar. 2018

18 citations recorded by the source corpus.

One of possible ways of obtaining continuous-space sentence representations is by training neural machine translation (NMT) systems. The recent attention mechanism however removes the single point in the neural network from which the source sentence representation can be extracted. We propose several variations of the attentive NMT architecture bringing this meeting point back. Empirical evaluation suggests that the better the translation quality, the worse the learned sentence representations serve in a wide range of classification and similarity tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, background): “However, this ﬁnding builds on the arguments in Cifka and Bojar (2018), which show that BLEU score does not correlate perfectly with semantics.”
Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features? (2020, methodology): “During test time, we force-decode the initial target language label before continuing the standard decoding process that generates the translation in the desired language. x 2 x 3 x T ... (Cífka and Bojar, 2018)”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “It should not be surprising that representational quality as measured by such systems is a poor indicator of translation quality (Cífka and Bojar, 2018).”