SYSTRAN's Pure Neural Machine Translation Systems

By Josep Maria Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng and 19 more. 2016

128 citations recorded by the source corpus.

Since the first online demonstration of Neural Machine Translation (NMT) by LISA, NMT development has recently moved from laboratory to production systems as demonstrated by several entities announcing roll-out of NMT engines to replace their existing technologies. NMT systems have a large number of training configurations and the training process of such systems is usually very long, often a few weeks, so role of experimentation is critical and important to share. In this work, we present our approach to production-ready systems simultaneously with release of online demonstrators covering a large variety of languages (12 languages, for 32 language pairs). We explore different practical choices: an efficient and evolutive open-source framework; data preparation; network architecture; additional implemented features; tuning for production; etc. We discuss about evaluation methodology, present our first findings and we finally outline further work. Our ultimate goal is to share our expertise to build competitive production systems for "generic" translation. We aim at contributing to set up a collaborative framework to speed-up adoption of the technology, foster further research efforts and enable the delivery and adoption to/by industry of use-case specific engines integrated in real production workflows. Mastering of the technology would allow us to build translation engines suited for particular needs, outperforming current simplest/uniform systems.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “, 2016a, 2017, 2018b) and for large-scale production systems (Wu et al., 2016; Zhou et al., 2016; Crego et al., 2016; Hassan et al., 2018).”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, background): “End-to-end Neural Machine Translation (NMT) [27, 2, 5] is an approach to machine translation that has rapidly gained adoption in many large-scale settings [31, 29, 6].”
OpenNMT: Open-Source Toolkit for Neural Machine Translation (2017, background): “Neural machine translation (NMT) is a new methodology for machine translation that has led to remarkable improvements, particularly in terms of human evaluation, compared to rule-based and statistical machine translation (SMT) systems (Wu et al., 2016; Crego et al., 2016).”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “The RNN-based NMT approach, or RNMT, was quickly established as the de-facto standard for NMT, and gained rapid adoption into large-scale systems in industry, e.g. Baidu (Zhou et al., 2016), Google (Wu et al., 2016), and Systran (Crego et al., 2016).”
Six Challenges for Neural Machine Translation (2017, background): “…has emerged as the most promising machine translation approach in recent years, showing superior performance on public benchmarks (Bojar et al., 2016) and rapid adoption in deployments by, e.g., Google (Wu et al., 2016), Systran (Crego et al., 2016), and WIPO (Junczys-Dowmunt et al., 2016).”
Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation (2018, background): “It dominated the recent WMT shared task (Bojar et al., 2017), and is used commercially (Wu et al., 2016; Crego et al., 2016; Junczys-Dowmunt et al., 2016).”
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, background): “Finally, another approach which shares the hard-decision made by lexically constrained decoding is the placeholder approach (Crego et al., 2016), wherein identiﬁable elements in the input are transformed to masks during preprocessing, and then replaced with their original source-language strings…”
Dynamic Data Selection for Neural Machine Translation (2017, background): “…or data (Wu et al., 2016), modifying the NMT network structure (Kalchbrenner et al., 2016), decreasing the number of parameters through knowledge distillation (Crego et al., 2016; Kim and Rush, 2016), or by boosting parts of the data that are ‘challenging’ to the NMT sys-tem (Zhang et al., 2016).”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “…describe related work, in terms of the parameter generator for NMT: • Pairwise: In the simple and commonly used pairwise NMT setting (Wu et al., 2016; Crego et al., 2016), the parameter generator would generate separate parameters, θ ( enc ) and θ ( dec ) , for each pair of source-target languages.”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, background): “Neural models have consistently shown top performance in shared evaluation tasks (Bojar et al., 2016; Cet-tolo et al., 2016) and are becoming the technology of choice for commercial MT service providers (Wu et al., 2016; Crego et al., 2016).”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “The success of neural machine translation, which has already been adopted by major industry players in machine translation (Wu et al., 2016; Crego et al., 2016), is often attributed to the advances in building and training recurrent networks as well as the availability of large-scale parallel…”
Training Neural Machine Translation to Apply Terminology Constraints (2019, methodology): “6 To our knowledge there is no existing work that 6Luong et al. (2015) and SYSTRANs Pure NMT system (Crego et al., 2016) are an exception to the constrained decoding approach as they replace entities with special tags that remain unchanged during translation and are replaced in a postprocessing…”
Competence-based Curriculum Learning for Neural Machine Translation (2019, background): “Zhang et al. (2016) propose training neural networks for NMT by focusing on hard examples, rather than easy ones.”
Trainable Greedy Decoding for Neural Machine Translation (2017, methodology): “We believe this favourable computational complexity makes the proposed method suitable for production-grade neural machine translation (Wu et al., 2016; Crego et al., 2016).”
Lexically Constrained Neural Machine Translation with Levenshtein Transformer (2020, methodology): “Crego et al. (2016) replaced entities with place-holders that remained unchanged during translation and placed them back in a post-processing step.”
Does Neural Machine Translation Benefit from Larger Context? (2017, background): “Attention-based neural machine translation, proposed by Bahdanau et al. (2014), has become de facto standard in recent years, both in academia (Bojar et al., 2016) and industry (Wu et al., 2016; Crego et al., 2016).”