Neural Machine Translation via Binary Code Prediction
By Yusuke Oda, Philip Arthur, Graham Neubig, Koichiro Yoshino, Satoshi Nakamura. 2017
17 citations recorded by the source corpus.
In this paper, we propose a new method for calculating the output layer in neural machine translation systems. The method is based on predicting a binary code for each word and can reduce computation time/memory requirements of the output layer to be logarithmic in vocabulary size in the best case. In addition, we also introduce two advanced approaches to improve the robustness of the proposed model: using error-correcting codes and combining softmax and binary codes. Experiments on two English-Japanese bidirectional translation tasks show proposed models achieve BLEU scores that approach the softmax, while reducing memory usage to the order of less than 1/10 and improving decoding speed on CPUs by x5 to x10.
Subfields: Artificial Intelligence, Molecular Biology, Efficiency & Compression, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Raj Dabre
- Atsushi Fujita
- Benjamin Heinzerling
- Kentaro Inui
- Chenhui Chu
- Anoop Kunchukuttan
- Danai Koutra
- Tara Safavi
In Efficiency & Compression
In Machine Translation
Citations in the corpus
- Softmax Tempering for Training Neural Machine Translation Models (2020, methodology): “We refer readers to orthogonal methods for speeding up NMT, such as weight pruning (See et al., 2016), quantization (Lin et al., 2016), and binarization (Courbariaux et al., 2017; Oda et al., 2017).”
- Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, methodology): “The path from the KD-tree’s root to a leaf can be represented as a binary code, which we use as training signal (Oda et al., 2017).”