Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

By Dario Stojanovski, Alexander M. Fraser. 2018

27 citations recorded by the source corpus.

Cross-sentence context can provide valuable information in Machine Translation and is critical for translation of anaphoric pronouns and for providing consistent translations. In this paper, we devise simple oracle experiments targeting coreference and coherence. Oracles are an easy way to evaluate the effect of different discourse-level phenomena in NMT using BLEU and eliminate the necessity to manually define challenge sets for this purpose. We propose two context-aware NMT models and compare them against models working on a concatenation of consecutive sentences. Concatenation models perform better, but are computationally expensive. We show that NMT models taking advantage of context oracle signals can achieve considerable gains in BLEU, of up to 7.02 BLEU for coreference and 1.89 BLEU for coherence on subtitles translation. Access to strong signals allows us to make clear comparisons between context-aware models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kellie Webster
2. Junlin Yang
3. Emily Pitler

In Computer Vision and Pattern Recognition

1. Kellie Webster
2. Junlin Yang
3. Emily Pitler

In Natural Language Processing

1. Kellie Webster
2. Junlin Yang
3. Emily Pitler
4. Adina Williams

Citations in the corpus

• Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “An oracle study found that augmenting an NMT system with perfect pronoun information could give up to 6 BLEU points improvement (Stojanovski and Fraser, 2018).”
• Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution (2019, background): “However, most state-of-art deep-learning-based resolvers utilize one-directional Transform-ers (Stojanovski and Fraser, 2018), limiting the ability to handle long-range inferences and the use of cataphors.”