Scalable Inference and Training of Context-Rich Syntactic Translation Models

By Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, Ignacio Thayer. 2006

475 citations recorded by the source corpus.

Statistical MT has made great progress in the last few years, but current translation models are weak on re-ordering and target language fluency. Syntactic approaches seek to remedy these problems. In this paper, we take the framework for acquiring multi-level syntactic translation rules of (Galley et al., 2004) from aligned tree-string pairs, and present two main extensions of their approach: first, instead of merely computing a single derivation that minimally explains a sentence pair, we construct a large number of derivations that include contextually richer rules, and account for multiple interpretations of unaligned words. Second, we propose probability estimates and a training procedure for weighting these rules. We contrast different approaches on real examples, show that our estimates based on multiple derivations favor phrasal re-orderings that are linguistically better motivated, and establish that our larger rules provide a 3.63 BLEU point increase over minimal rules.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Recurrent Neural Network Grammars (2016, background): “Previous work in machine translation has showed that conditional syntactic models can function quite well without the computationally expensive marginalization process at decoding time (Galley et al., 2006; Gimpel and Smith, 2014).”
Does String-Based Neural MT Learn Source Syntax? (2016, background): “Non-neural machine translation systems consist chieﬂy of phrase-based systems (Koehn et al., 2003) and syntax-based systems (Galley et al., 2004; Galley et al., 2006; DeNeefe et al., 2007; Liu et al., 2011; Cowan et al., 2006), the latter of which adds syntactic information to source side…”
Six Challenges for Neural Machine Translation (2017, background): “…we note that there are other statistical machine translation approaches such as hierarchical phrase-based models (Chiang, 2007) and syntax-based models (Galley et al., 2004, 2006) that have been shown to give superior performance for language pairs such as Chinese– English and German–English.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “Table 1 shows that for 4 low-resource languages, a standard string-to-tree statistical MT system (SBMT) (Galley et al., 2004; Galley et al., 2006) strongly outperforms NMT, even when NMT uses the state-of-the-art lo-cal attention plus feed-input techniques from Lu-ong et al. (2015a).”
EU-BRIDGE MT: Combined Machine Translation (2014, background): “Composed rules (Galley et al., 2006) are extracted in addition to minimal rules, but only up to the following limits: at most twenty tree nodes per rule,”
11,001 New Features for Statistical Machine Translation (2009, methodology): “We follow Galley et al. (2006) in allowing unaligned Chinese words to participate in multiple translation rules, and in collecting larger rules composed of minimal rules.”
On the Impact of Various Types of Noise on Neural Machine Translation (2018, methodology): “…systems as our SMT paradigm, we note that there are other statistical machine translation approaches such as hierarchical phrase-based models (Chiang, 2007) and syntax-based models (Galley et al., 2004, 2006) that may have better performance in certain language pairs and in low resource conditions.”
Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes (2015, neutral): “WASP casts semantic parsing as a syntax-based statistical machine translation (SMT) task, where a synchronous context-free grammar (SCFG) (Wu, 1997; Chiang, 2005; Galley et al., 2006) is used to model the translation of natural language into a formal meaning representation.”
Edinburgh’s Syntax-Based Systems at WMT 2015 (2014, methodology): “SCFG rules were extracted from the wordaligned parallel data using the Moses implementation (Williams and Koehn, 2012) of the GHKM algorithm (Galley et al., 2004; Galley et al., 2006).”
On the Linguistic Representational Power of Neural Machine Translation Models (2019, background): “For more than a decade of research in machine translation, a tremendous amount of effort has been put into syntax-based machine translation (Yamada and Knight (2002); Chiang (2005), Galley et al. (2006), Zhang et al.”
Constructing Parallel Corpora for Six Indian Languages via Crowdsourcing (2012, background): “3Weuse hierarchical to denote translation grammars that use only a single nonterminal (Chiang, 2007), in contrast to syntactic systems, which make use of linguistic annotations (Zollmann and Venugopal, 2006; Galley et al., 2006).”
First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests (2009, background): “…space considered (for a given input) by a mono-lingual parser or a tree-based translation system, e.g., tree to string (Quirk et al., 2005; Liu et al., 2006), string to tree (Galley et al., 2006), tree to tree (Eisner, 2003), or string to string with latent tree structures (Chiang, 2007).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “String-to-tree approaches map a source sentence to a target tree, and have proved to be quite successful [53, 114, 115, 305, 347, 348]; tree-to-string approaches map a source tree to a target sentence [150, 213, 216, 256, 257, 280]; and tree-to-tree map from a source tree to a target tree [91, 100, ”
Factored Translation Models (2007, background): “The goal of integrating syntactic information into the translation model has prompted many researchers to pursue tree-based transfer models (Wu, 1997; Alshawi et al., 1998; Yamada and Knight, 2001; Melamed, 2004; Menezes and Quirk, 2005; Galley et al., 2006), with increasingly encouraging results.”
Statistical machine translation (2010, unknown): “Tree-adjoining grammars (TAG, tag (), see also the review by Joshi et al. (1990)) start from a set of initial trees and use tree substitution and tree adjunction to derive a tree.”
A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation (2015, methodology): “Speciﬁ-cally, we work with a string-to-tree model with GHKM-style rules (Galley et al., 2006), and a relational dependency language model (Sennrich, 2015).”
Retrieval-Based Neural Code Generation (2018, methodology): “In Galley et al. (2006), subtrees of the input sentence’s parse tree are associated with corresponding words in the output sentence.”
A Tree-based Decoder for Neural Machine Translation (2018, background): “Motivated by the success of adding syntactic information to Statistical Machine Translation (SMT) (Galley et al., 2004; Menezes and Quirk, 2007; Galley et al., 2006), recent works have established that explicitly leveraging syntactic information can improve NMT 1 Our code is available at…”
NiuTrans: An Open Source Toolkit for Phrase-based and Syntax-based Machine Translation (2012, methodology): “For unaligned words, we attach them to all nearby rules, instead of using the most likely attachment as in (Galley et al., 2006).”