Evaluating Discourse Phenomena in Neural Machine Translation

By Rachel Bawden, Rico Sennrich, Alexandra Birch, Barry Haddow. 2017

287 citations recorded by the source corpus.

For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models’ ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “gy/W13-3300 6http://opus.nlpl.eu/TED2013.php 7http://opus.nlpl.eu/OpenSubtitles.php second person pronouns and noisy alignment. High quality targeted datasets for discourse phenomena were explored in Bawden et al. (2018), however the released corpus was a small French→English test set. In this paper”
Evaluating Gender Bias in Machine Translation (2019, background): “These include challenge sets for language-speciﬁc idiosyncrasies (Isabelle et al., 2017), discourse phenomena (Bawden et al., 2018), pronoun translation (M¨uller et al., 2018; Webster et al., 2018), or coreference and multiword expressions (Burchardt et al., 2017).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “, 2018; Belinkov and Bisk, 2018). Gao et al. (2018) defined scoring functions to identify tokens to modify.”
Stress Test Evaluation for Natural Language Inference (2018, background): “Unlike work on manual construction of small adversarial evaluation sets for various NLP tasks (Levesque, 2014; Mahler et al., 2017; Staliūnaite and Bonfil, 2017; Isabelle et al., 2017; Belinkov and Bisk, 2017; Bawden et al., 2017), our work focuses on a”
Context-Aware Neural Machine Translation Learns Anaphora Resolution (2018, background): “tion (Hardmeier et al., 2015; Miculicich Werlen and Popescu-Belis, 2017; Bawden et al., 2018).”
Pronoun Translation in English-French Machine Translation: An Analysis of Error Types (2018, background): “The first (LIMSI) is the s-hier system described in paper [18].”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, background): “…phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018, inter alia ) for document-level MT.”
Neural Machine Translation: A Review (2019, methodology): “use source context only [595, 596, 600, 605, 606], target context only [597, 599], or both [527, 598, 601, 602, 604].”
Data augmentation using back-translation for context-aware neural machine translation (2019, methodology): “…↔ fr and ja → en datasets, • developed a new specialized test set for evaluating ja → en context-aware translation, and • conﬁrmed that the data augmentation improves context-aware translation through the existing en → fr (Bawden et al., 2018) and our specialized test set for ja → en translation.”
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, background): “For example, these cover things like grammatical agreement (Sennrich, 2017), translation of pronouns or other discourse-sensitive phenomena (Müller et al., 2018; Bawden et al., 2018), or diagnostic tests for a variety of different phenomena (Isabelle et al.”
DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation (2019, background): “The second (2TO2), is trained to translate sentences in the context of the previous sentence, as in (Tiedemann and Scherrer, 2017) and (Bawden et al., 2018).”
Improving the Transformer Translation Model with Document-Level Context (2018, background): “…both for conventional SMT (Gong et al., 2011; Hardmeier et al., 2012; Xiong et al., 2013a,b; Garcia et al., 2014) and NMT (Jean et al., 2017; Kuang et al., 2017; Tiedemann and Scher-rer, 2017; Wang et al., 2017; Maruf and Haffari, 2018; Bawden et al., 2018; Tu et al., 2018; Voita et al., 2018).”
Competence-based Curriculum Learning for Neural Machine Translation (2019, background): “These are just two examples of difﬁculty metrics, and it is easy to conceive of other metrics such as the occurrence of homographs (Liu et al., 2018) or context-sensitive words (Bawden et al., 2018), the examination of which we leave for future work.”
Modeling Recurrence for Transformer (2019, unknown): “Bawden et al. (2018) and Voita et al.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Following work in NLP [196] and machine translation [178], a number of such suites have been manually constructed to evaluate neural machine translation performance on a range of linguistic phenomena [23, 49, 161].”
Putting Machine Translation in Context with the Noisy Channel Model (2019, background): “Recent studies (Tiedemann & Scherrer, 2017; Bawden et al., 2018, inter alia ) have shown that exploiting document-level context improves translation performance, and in particular improves lexical consistency and coherence of the translated text.”
GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, background): “The discourse phenomena, such as pronominal anaphora and lexical consistency will be neglected by sentence-level MT systems, which makes it far worse than human translations [21, 22].”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, methodology): “…context-aware NMT models to leverage either local context, e.g., previous sentences (Jean et al., 2017; Wang et al., 2017; Zhang et al., 2018; Bawden et al., 2018; Voita et al., 2018, 2019b; Yang et al., 2019), or entire document (Maruf and Haffari, 2018; Mace and Servan, 2019; Maruf et al.,…”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, unknown): “…when additional information is necessary for the output to be well-formed (e.g. translating anaphoric pronouns into a language that requires agreement with a coreferent) (Hardmeier, 2012; Libovick´y and Helcl, 2017; Bawden et al., 2018; Voita et al., 2018; Lopes et al., 2020; Nayak et al., 2022).”