Depthwise Separable Convolutions for Neural Machine Translation

By Lukasz Kaiser, Aidan N. Gomez, François Chollet. 2017

314 citations recorded by the source corpus.

Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Universal Transformers (2018, methodology): “…in parallel using a self-attention mechanism (Parikh et al., 2016; Lin et al., 2017), followed by a transformation (shared across all positions and time-steps) consisting of a depth-wise separable convolution (Chollet, 2016; Kaiser et al., 2017) or a position-wise fully-connected layer (see Fig 1).”
Non-Autoregressive Neural Machine Translation (2017, background): “While several recently proposed models avoid recurrence at train time by leveraging convolutions (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2017) or self-attention (Vaswani et al.”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “We add sinusoidal position embeddings to encode the absolute position of each word in the sequence (Kaiser et al., 2017; Vaswani et al., 2017).”
Tensor2Tensor for Neural Machine Translation (2018, methodology): “The same technique was improved in a number of neural translation models recently, including Gehring et al. (2017) and Kaiser et al. (2017). continues in applying the model in more domains and exploring the space of self-attention mechanisms.”
Efficient Transformers: A Survey (2020, background): “To some extent, Linformer is reminiscent of depth-wise convolutions [35].”
One Model To Learn Them All (2017, methodology): “For convolution operations, we use depthwise separable convolutions, introduced in [5], in a way similar to [15].”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, background): “Depthwise separable convolution (also referred to as separable convolution) has proven effective in image processing tasks [32], [33] and neural machine translation tasks [39].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Convolutional Encoder: Inspired by the recent trend of replacing recurrent networks with CNNs [6, 15, 41] and Transformers [37, 7] for sequence modeling, we use positional encoding (PE) [37], CNNs, and layer normalization [3] to build our basic encoding block.”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “The same technique was improved in a number of neural translation models recently, including (Gehring et al., 2017) and (Kaiser et al., 2017).”
Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification (2017, neutral): “Recently, convolutional sequence-to-sequence networks have been successfully applied to machine translation tasks [5, 8].”
EfficientPS: Efficient Panoptic Segmentation (2020, background): “Our proposed backbone is substantially more efficient as well as effective than its popular counterparts (He et al, 2016; Kaiser et al, 2017; Xie et al, 2017).”
UvA-DARE (Digital Academic Repository) Learning to Transform, Combine, and Reason in Open-domain Question Answering (unknown): “…feed-forward network, which is applied to each position separately and identically: This stack of N Transformer Encoder blocks is followed by a depth-wise separable convolution [9, 20] and then a pooling function to get a single vector representation for the whole document (or the question).”
The University of Cambridge’s Machine Translation Systems for WMT18 (2018, methodology): “On English-German and German-English news-test2014 we compute cased BLEU scores with Moses’ multi-bleu.pl script on tokenized output to be comparable with prior work (Wu et al., 2016; Kaiser et al., 2017; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018).”
Scaling Up Online Speech Recognition Using ConvNets (2020, methodology): “Depth-wise separable convolutions [18] have been used in domains such as computer vision [19] and machine translation [20] yielding dramatic reductions in model size and computational throughput while maintaining accuracy.”
Dense Information Flow for Neural Machine Translation (2018, methodology): “, 2017) uses self-attention function before a two-layer position-wise feedforward networks, and SliceNet (Kaiser et al., 2017) uses a combination of ReLU, depthwise separable convolution, and layer normalization.”
Neural Machine Translation with Byte-Level Subwords (2019, methodology): “We propose to use either a depth-wise convolutional layer (Kaiser, Gomez, and Chollet 2017) or a bidirectional recurrent layer with gated recurrent units (Cho et al. 2014, GRU,) Decoding with Byte-Level Subwords While any sentence can be represented as a byte sequence, the converse is, however, not…”
Lite Transformer with Long-Short Range Attention (2020, background): “For instance, researchers have proposed highly-efficient convolution-based models (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2018; Wu et al., 2019b).”
On NMT Search Errors and Model Errors: Cat Got Your Tongue? (2019, background): “Table 2: ∗: The recurrent LSTM, the convolutional SliceNet (Kaiser et al., 2017), and the Transformer-Big systems are strong baselines from a WMT’18 shared task submission (Stahlberg et al.”