DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

By Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei. 2021

92 citations recorded by the source corpus.

While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG). NLG tasks are often based on the encoder-decoder framework, where the pretrained encoders can only benefit part of it. To reduce this gap, we introduce DeltaLM, a pretrained multilingual encoder-decoder model that regards the decoder as the task layer of off-the-shelf pretrained encoders. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. To take advantage of both the large-scale monolingual data and bilingual data, we adopt the span corruption and translation span corruption as the pre-training tasks. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks, including machine translation, abstractive text summarization, data-to-text, and question generation. The code and pretrained models are available at \url{https://aka.ms/deltalm}.

Subfields: Artificial Intelligence, Large Language Models, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Wenhui Wang
6. Saksham Singhal
7. Hangbo Bao
8. Zewen Chi

In Large Language Models

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Wenhui Wang
6. Hangbo Bao
7. Xia Song
8. Zewen Chi

In Machine Translation

1. Li Dong
2. Furu Wei

Citations in the corpus

• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “Pre-training with Transformer backbone networks has substantially advanced the state of the art across natural language processing (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2019; Dong et al., 2019; Raffel et al., 2020; Bao et al., 2020; Conneau and Lample, 2019; Con”
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, background): “∆ LM (Ma et al., 2021) proposes to align pretrained multilingual encoders to improve cross-lingual sequence-to-sequence pre-training.”
• DeepNet: Scaling Transformers to 1,000 Layers (2022, background): “Transformers have achieved success across many fields, including machine translation [1], [61], language modelling [2], [4], [5], [45], speech recognition [62], vision pre-training [31], [53], [54] and vision-language pre-training [63].”