Neural Machine Translation in Linear Time

By Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron van den Oord, Alex Graves, Koray Kavukcuoglu. 2016

588 citations recorded by the source corpus.

We present a novel neural network for processing sequences. The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. To address the differing lengths of the source and the target, we introduce an efficient mechanism by which the decoder is dynamically unfolded over the representation of the encoder. The ByteNet uses dilation in the convolutional layers to increase its receptive field. The resulting network has two core properties: it runs in time that is linear in the length of the sequences and it sidesteps the need for excessive memorization. The ByteNet decoder attains state-of-the-art performance on character-level language modelling and outperforms the previous best results obtained with recurrent networks. The ByteNet also achieves state-of-the-art performance on character-to-character machine translation on the English-to-German WMT translation task, surpassing comparable neural translation models that are based on recurrent networks with attentional pooling and run in quadratic time. We find that the latent alignment structure contained in the representations reflects the expected alignment between the tokens.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “…English to German translation we compare to the following prior work: Luong et al. (2015) is based on a four layer LSTM attention model, ByteNet (Kalchbrenner et al., 2016) propose a convolutional model based on characters without attention, with 30 layers in the encoder and 30 layers in the…”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “ByteNet [19], which is related but uses layers of dilated convolutions rather than attention, showed promising results on byte level language modeling.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016);…”
Self-Attention with Relative Position Representations (2018, neutral): “, 2014), convolution (Gehring et al., 2017; Kalchbrenner et al., 2016), attention (Vaswani et al.”
Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “7L LSTM (Graves, 2013) 1.67 LN HyperNetworks Ha et al. (2016) 1.34 LN HM-LSTM Chung et al. (2016) 1.32 ByteNet (Kalchbrenner et al., 2016) 1.31 RHN Zilly et al. (2017) 1.27 mLSTM Krause et al. (2016) 1.24 64L Transf.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
Non-Autoregressive Neural Machine Translation (2017, background): “While several recently proposed models avoid recurrence at train time by leveraging convolutions (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2017) or self-attention (Vaswani et al.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “…begun to successfully model high dimensional raw observations from complex real-world datasets, from natural images and videos, to audio signals and natural language (Karras et al., 2017; Kalchbrenner et al., 2016b; van den Oord et al., 2016a; Kalchbrenner et al., 2016a; Vaswani et al., 2017).”
Efficient Neural Audio Synthesis (2018, background): “Convolutional sequence models (Kalchbrenner et al., 2016) achieve excellent performance in speech synthesis (Wang et al., 2017), yet their architecture tends to be deep and narrow requiring a long chain of layers to be executed for each sample.”
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “One way of allowing the network to model structure at multiple resolutions is to introduce dilated convolutions into the model, as proposed by van den Oord et al. (2016a) and Kalchbrenner et al. (2016b).”
A Convolutional Encoder Model for Neural Machine Translation (2016, background): “Concurrently to our work, (Kalchbrenner et al., 2016) have introduced convolutional translation models without an explicit attention mechanism but their approach does not yet result in state-ofthe-art accuracy. (Lamb and Xie, 2016) also proposed a multi-layer CNN to generate a fixed-size encoder rep”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, background): “Outside of vision, residuals have displayed impressive performance in audio generation [29] and neural machine translation [14], ResNets are built out of modules called residual blocks, which have the following form: where F , a function called the residual function, is typically a shallow neural…”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “…in sequence modeling through recurrent neural networks (RNN; Sutskever et al. 2014; Bahdanau et al. 2015; Wu et al. 2016), convolutional networks (CNN; Kalchbrenner et al. 2016; Gehring et al. 2016; 2017; Kaiser et al. 2017) and self-attention models (Paulus et al., 2017; Vaswani et al., 2017).”
Language Modeling with Gated Convolutional Networks (2016, background): “Later, Kalchbrenner et al. (2016) extended this mechanism with additional gates for use in translation and character-level language modeling.”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “Furthermore, advances in convolutional architecture design may be applied to tasks outside of the image domain [38, 20].”
Tensor2Tensor for Neural Machine Translation (2018, methodology): “Doing so requires a stack of O ( n/k ) convolutional layers in the case of contiguous kernels, or O ( log k ( n )) in the case of dilated convolutions (Kalchbrenner et al., 2016), increasing the length of the longest paths between any two positions in the network.”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, background): “Well known examples are ByteNet (Kalchbrenner et al., 2016) and ConvS2S (Gehring et al.”