Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

By Jungo Kasai, Nikolaos Pappas, James Cross, Noah A. Smith. 2020

156 citations recorded by the source corpus.

Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to the latter, where generation is sequential, the former allows generation to be parallelized across target token positions. Some of the latest non-autoregressive models have achieved impressive translation quality-speed tradeoffs compared to autoregressive baselines. In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a single-layer autoregressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol for future research toward fast, accurate machine translation. Our code is available at https://github.com/jungokasai/deep-shallow.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, methodology): “…are reported averaged over 1000 samples with a batch size of 50, evaluated on FFHQ while using the same number of trainable parameters (800m) for all AR models. signiﬁcantly reduce the number of decoder layers and thus also decrease autoregressive decoding speed while maintaining high quality [32].”
Scaling Laws for Neural Machine Translation (2021, background): “This is contrary to the usual practice; due to latency considerations, many practitioners train NMT models with deep encoders and shallow decoders (Kasai et al., 2021).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “(3)The benefits of such re-scaling have also been observed for subword-based encoder-decoder neural machine translation models (Devlin, 2017; Kasai et al., 2021).”
HyperTree Proof Search for Neural Theorem Proving (2022, unknown): “[42] in the context of machine translation.”
Big Little Transformer Decoder (2023, background): “However, non-autoregressive decoding suffers from text generation quality issues due to its assumption of conditional in-dependence between output tokens [27].”
Continuous diffusion for categorical data (2022, methodology): “Kasai et al. (2020b) questioned the speed advantage of non-AR models in machine translation by comparing them to a shallow-decoder AR baseline.”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, background): “, 2016a), model ensembling, and deep encoders (Kasai et al., 2021a).”
Scalable and Efficient MoE Training for Multitask Multilingual Models (2021, methodology): “For better inference efﬁciency, we use deeper encoders and shallower decoders architecture presented in (Kim et al., 2019) and (Kasai et al., 2020).”
Confident Adaptive Language Modeling (2022, background): “This is especially pronounced in the autoregressive decoding process where the full stack of Transformer layers is repeatedly computed for each output token [37; 40; 86].”
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, methodology): “This encoder-favored modeling has been demonstrated by previous work to improve latency with little generation quality degradation (Kasai et al., 2021; Sun et al., 2021; Ge et al., 2022a).”