Predicting Target Language CCG Supertags Improves Neural Machine Translation

By Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, Alexandra Birch. 2017

82 citations recorded by the source corpus.

Neural machine translation (NMT) models are able to partially learn syntactic information from sequential lexical information. Still, some complex syntactic phenomena such as prepositional phrase attachment are poorly modeled. This work aims to answer two questions: 1) Does explicitly modeling target language syntax help NMT? 2) Is tight integration of words and syntax better than multitask training? We introduce syntactic information in the form of CCG supertags in the decoder, by interleaving the target supertags with the word sequence. Our results on WMT data show that explicitly modeling target-syntax improves machine translation quality for German->English, a high-resource pair, and for Romanian->English, a low-resource pair and also several syntactic phenomena including prepositional phrase attachment. Furthermore, a tight coupling of words and syntax improves translation quality more than multitask training. By combining target-syntax with adding source-side dependency labels in the embedding layer, we obtain a total improvement of 0.9 BLEU for German->English and 1.2 BLEU for Romanian->English.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, positive): “This approach is inspired by similar approaches in phrase-based MT (Bojar, 2007; Toutanova et al., 2008; Fraser, 2009) and was developed in parallel to recent similar experiments with two-step neural MT (Nadejde et al., 2017; Tamchyna et al., 2017).”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “…et al., 2015; Dong et al., 2015), multi-source translation (Zoph and Knight, 2016; Firat et al., 2016b; Garmash and Monz, 2016), multimodal translation (Caglayan et al., 2016) and linguistic knowledge guided translation (Nadejde et al., 2017; Aharoni and Goldberg, 2017; Eriguchi et al., 2017).”
Towards String-To-Tree Neural Machine Translation (2017, background): “Comparing our approach to other syntax aware NMT models like Eriguchi et al. (2017) and Nadejde et al. (2017) may also be of interest.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “170ne remedy may be to interleave words and tags [250].”
Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features? (2020, background): “There have been indications that explicitly modeling linguistic information can help performance of neural machine translation (NMT) models (Aharoni and Goldberg, 2017; Nadejde et al., 2017).”
Neural Machine Translation: A Review (2019, methodology): “popular approach is to retain the sequence-to-sequence architecture and linearize the tree structures, for example using bracket expressions [526, 562–564], sequences of rules [329], or CCG supertags [565].”
A Tree-based Decoder for Neural Machine Translation (2018, methodology): “…CCG: a syntax-aware translation model that interleaves Combinatory Cate-gorial Grammar (CCG) tags with words on the target side of a seq2seq model (Nadejde et al., 2017); 3) CCG-null: the same model with CCG, but all syntactic tags are replaced by a null tag; and 4) LIN: a standard seq2seq model…”