Sockeye: A Toolkit for Neural Machine Translation
By Felix Hieber, Tobias Domhan, Michael J. Denkowski, David Vilar, Artem Sokolov, Ann Clifton, Matt Post. 2017
217 citations recorded by the source corpus.
We describe Sockeye (version 1.12), an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). Sockeye is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNet, the toolkit offers scalable training and inference for the three most prominent encoder-decoder architectures: attentional recurrent neural networks, self-attentional transformers, and fully convolutional networks. Sockeye also supports a wide range of optimizers, normalization and regularization techniques, and inference improvements from current NMT literature. Users can easily run standard training recipes, explore different model settings, and incorporate new ideas. In this paper, we highlight Sockeye's features and benchmark it against other NMT toolkits on two language arcs from the 2017 Conference on Machine Translation (WMT): English-German and Latvian-English. We report competitive BLEU scores across all three architectures, including an overall best score for Sockeye's transformer implementation. To facilitate further comparison, we release all system outputs and training scripts used in our experiments. The Sockeye toolkit is free software released under the Apache 2.0 license.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Philipp Koehn
- Jianfeng Gao
- Kyunghyun Cho
- Felix Stahlberg
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Felix Stahlberg
- Michael Auli
- Luc Van Gool
- Philipp Koehn
- Dacheng Tao
- Kyunghyun Cho
- Trevor Darrell
- Xiaogang Wang
In Natural Language Processing
- Graham Neubig
- Philipp Koehn
- Michael Auli
- Luke S. Zettlemoyer
- Rico Sennrich
- Felix Stahlberg
- Ondrej Bojar
- Kyunghyun Cho
In Machine Translation
- Philipp Koehn
- Ondrej Bojar
- Marine Carpuat
- Matīss Rikters
- David Vilar
- Matt Post
- Kevin Duh
- Rico Sennrich
Citations in the corpus
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2018) and Sockeye (Hieber et al., 2018) focus on production-readiness.”
- Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “2.4.25 RWTH (Schamper et al., 2018) All systems submitted by RWTH Aachen for German to English are based on the Transformer architecture implemented in Sockeye.”
- Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (2018, methodology): “We aim to create a level playing field for the comparison by training with the same toolkit, Sockeye (Hieber et al., 2017) which is based on MXNet (Chen et al., 2015).”
- Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “We implemented5 both EWC and MCL in Sockeye (Hieber et al., 2017).”
- Iterative Back-Translation for Neural Machine Translation (2018, methodology): “The difference here is on the NMT toolkit used — we opted to use Amazon’s Sockeye (Hieber et al., 2017).”
- Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, methodology): “The inference procedure in S OCKEYE maximizes the length-normalized version of the sentence’s log probability.”
- How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures (2018, methodology): “All experiments were run with an adapted version of S OCKEYE (Hieber et al., 2017), which can parse arbitrary model definitions that are expressed in the language described in Section 2.3.”
- Debugging Neural Machine Translations (2018, methodology): “In addition to Nematus, Neural Monkey and Marian 3 [6], we have also added out-of-the-box support for working with attention alignments from OpenNMT and Sockeye 4 [5] frameworks.”
- Tilde’s Machine Translation Systems for WMT 2017 (2018, methodology): “In order to train the NMT systems, we used the Nematus (Sennrich et al., 2017b) (for MLSTM models) and Sockeye (Hieber et al., 2017) (for Transformer models) toolkits.”
- Zero-Resource Neural Machine Translation with Monolingual Pivot Data (2019, methodology): “We use the Sockeye toolkit (Hieber et al., 2017) to run all experiments.”
- Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation (2019, methodology): “Sockeye Adapted have smarter motivations . and their parents also taught their children how to do it .”
- An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation (2018, methodology): “We use the Sockeye (Hieber et al., 2017) toolkit, which is based on MXNet (Chen et al., 2015), to train models.”
- Graph-to-Sequence Learning using Gated Graph Neural Networks (2018, methodology): “To tackle these issues, they propose Gated Graph Neural Networks, which extend these architectures with gating mechanisms 1 Our implementation uses MXNet (Chen et al., 2015) and is based on the Sockeye toolkit (Hieber et al., 2017).”
- Guided Generation of Cause and Effect (2020, methodology): “We use Sockeye [Hieber et al. , 2017] et al. , 2016 ] to reduce vocabulary size.”
- RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition (2018, methodology): “Again by our speed comparisons and the comparisons performed by the Sockeye authors (Hieber et al., 2017), one can conclude that RETURNN is very competitive in terms of speed, but is much more flexible.”
- An Empirical Exploration of Curriculum Learning for Neural Machine Translation (2018, methodology): “In practice, training protocols randomize the order of sentence pairs in the training corpus (Sennrich et al., 2017; Hieber et al., 2017).”
- Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation (2019, methodology): “All models are trained using the Sockeye seq2seq framework (Hieber et al., 2017) and averaged over 3 trials.”
- Learning to Segment Inputs for NMT Favors Character-Level Processing (2018, unknown): “For standard cross-entropy training the target history in ACT-DEC ˆ y nt (lines 22 and 24) is replaced by the target symbols at the correspond-as e.g. in (4) of (Hieber et al., 2017). ing positions y nt (teacher forcing).”