Revisiting Character-Based Neural Machine Translation with Capacity and Compression

By Colin Cherry, George F. Foster, Ankur Bapna, Orhan Firat, Wolfgang Macherey. 2018

101 citations recorded by the source corpus.

Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for character-level NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Our results here agree with existing literature (Cherry et al., 2018; Kreutzer and Sokolov, 2018) suggesting that using smaller sub-word tokens, as is the case for smaller vocabularies, performs better in low resource settings due to improved generalization.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “(2019) for speech recognition and synthesis, and many authors for machine translation (Lee et al., 2017; Costa-jussà et al., 2017; Cherry et al., 2018; Shaham and Levy, 2021).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “Cherry et al. (2018) show that trained long enough character-level models tend to have better quality, but it comes with the increase of computational cost for both training and inference.”
Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, unknown): “More recently, some studies notice this problem and propose some practical solutions (Kreutzer and Sokolov, 2018; Cherry et al., 2018; Chen et al., 2019; Salesky et al., 2020).”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “Furthermore, when trying to use the HM-RNN for NMT, Cherry et al. (2018) report that it took a lot of ﬁxing to get it to train at all; its performance on the task was competitive but not superior.”
Learning to Segment Inputs for NMT Favors Character-Level Processing (2018, unknown): “Concurrently to our work, Cherry et al. (2018) adapt hierarchical multi-scale RNNs (Chung et al., 2017) to NMT and compare them to several compression algorithms for character-based NMT. Similar to our work, they focus on the encoder and come to the same conclusion: deep recurrent models at…”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, background): “In related work, Cherry et al. (2018) have shown that, given deep encoders and decoders, character-level models can outperform other subword segmentations.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, methodology): “This is consistent with previous work supporting character level segmentation for improving NMT of LRL [13, 14].”
Lattice-Based Transformer Encoder for Neural Machine Translation (2019, background): “Meanwhile, the self-attention mechanism is good at learning the dependency between characters in parallel, which can partially compare and learn information from multiple segmentations (Cherry et al., 2018).”
Neural Machine Translation without Embeddings (2020, background): “Although earlier results on LSTM-based models show that character tokenization can outperform subword tokenization (Cherry et al., 2018), recent literature shows that the same does not hold for transformers (Gupta et al., 2019; Wang et al., 2019; Gao et al., 2020).”
Multilingual Neural Machine Translation With Soft Decoupled Encoding (2019, methodology): “Several prior works have utilized character level embeddings for machine translation (Cherry et al., 2018; Lee et al., 2017; Ataman & Federico, 2018), language modeling (Kim et al., 2016; J´ozefowicz et al., 2016), and semantic parsing (Yih et al., 2014).”
Neural Machine Translation: A Review (2019, methodology): “− Character-based NMT computationally more expensive than subword-based NMT (Cherry et al., 2018).”
Neural Machine Translation with Byte-Level Subwords (2019, background): “2016) proposed building word representations from character ones; (Chung, Cho, and Bengio 2016) removed the restriction of word boundaries and directly learned decoding in character level; (Lee, Cho, and Hofmann 2017) further extended it to a fully character-level model in a multilingual setting; (C”