Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation

By Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, Noah A. Smith. 2020

51 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “Recent studies show that encoders are more important for bilingual machine translation (Wu et al., 2019; Kasai et al., 2020), however, these studies are based on systems modeling only a single language direction compared to our setting.”
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “To reduce the computational cost for decoding, we propose to use a shallow decoder, which has proven to be an effective strategy (Kasai et al., 2020; Li et al., 2021) in neural machine translation (NMT), instead of using the Transformer with balanced encoder-decoder depth as the previous…”
Random Feature Attention (2021, methodology): “As noted by Katharopoulos et al. (2020) and Kasai et al. (2021), if we had used mini-8 batches as large as the hardware allows, R FA could have achieved a more signiﬁcant speed gain.”
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “However, recent studies (Kasai et al., 2020b) indicated that iterative NAT models seem to lose the speed advantage after carefully tuning the layer allocation of AT models.”
Pre-trained Summarization Distillation (2020, methodology): “Distillation for Seq2Seq models has primarily used pseudo-labeling and produces strong results on machine translation, as shown in Kasai et al. (2020), Junczys-Dowmunt (2019), and Sun et al. (2019).”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, methodology): “Moreover, due to the sequential nature of the autoregressive decoding (Kasai et al., 2020; Chen et al., 2018), the added communication cost from model parallel decoders gets multiplied by the number of decoding steps.”
Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021, unknown): “Shallow Decoders (SD): Shallow decoder models simply reduce the decoder depth and increase the encoder depth in response to the observation that decoding latency is proportional to the number of decoder layers (Kim et al., 2019; Miceli Barone et al., 2017; Wang et al., 2019; Kasai et al., 2020).”
On Learning Universal Representations Across Languages (2020, background): “Previous work have veriﬁed the use of deep encoders and shallow decoders to improve translation speed (Kim et al., 2019; Kasai et al., 2020) and accuracy (Miceli Barone et al., 2017; Wang et al., 2019b).”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, methodology): “Second, a strong encoder is important for NLG, according to the empirical studies of the previous work (Kasai et al., 2020).”