Cross-Lingual Natural Language Generation via Pre-Training

By Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, Heyan Huang. 2019

142 citations recorded by the source corpus.

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at this https URL.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “We are also interested in extending UNILM to support cross-lingual tasks [6].”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Following the recent trend of using multi-lingual pre-trained models for NLP tasks, such as Multilin-gual BERT (M-BERT) (Pires et al., 2019) 1 or XLM (Lample and Conneau, 2019), Chi et al. (2019) proposed to ﬁne-tune the models for summarization on English training data.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “Pre-training with Transformer [45] backbone networks has substantially advanced the state of the art across natural language processing [34, 10, 28, 22, 11, 36, 1, 7, 8, 4–6, The approaches of vision-language pre-training can be divided into two categories.”
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (2020, neutral): “2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2019; Lan et al., 2019; Raffel et al., 2019; Chi et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “In addition, XNLG optimizes parameters in a two-stage manner.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “The recent success of crosslingual language model pre-training (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020a; Chi et al., 2020; Liu et al., 2020) significantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and qu”
Pre-trained models for natural language processing: A survey (2020, methodology): “Transformer MASS [154], BART [98] XNLG [19], mBART [112]”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, methodology): “For fairly comparison, we implement xDAE in same code base and use same pre-training languages as XNLG.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, background): “XNLG (Chi et al., 2020) presents cross-lingual masked language modeling and cross-lingual auto-encoding for cross-lingual natural language generation, and achieves the cross-lingual transfer for NLG tasks.”
Models and Datasets for Cross-Lingual Summarisation (2022, background): “, building monolingual summarisation systems for different languages) has been gaining momentum (Chi et al., 2020b; Scialom et al., 2020).”
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (2020, methodology): “Similar techniques are used in the literature to control the style of the output (Keskar et al., 2019; Scialom et al., 2020; Chi et al., 2020).”
QAmeleon: Multilingual QA with Only 5 Examples (2022, methodology): “Lee et al. (2018) propose to use automatically translated high-confidence QA examples for training, while other approaches (Kumar et al., 2019; Chi et al., 2020) only generate questions and require supervised training data in the target language.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, unknown): “Besides monolingual pretrained models, multilingual pretrained models (Devlin et al., 2018; Lample and Conneau, 2019; Chi et al., 2019; Conneau et al., 2019; Chi et al., 2020) also advance the state-of-the-art on cross-lingual understanding and generation.”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, methodology): “For the bilingual corpus, we use the same corpus as I NFO XLM (Chi et al., 2020b), including MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2017), OPUS (Tiedemann, 2012), and WikiMatrix (Schwenk et al., 2019) We use a transformer-encoder (Vaswani et al., 2017) as the backbone of…”
VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation (2020, methodology): “Chi et al. (2020a) ﬁrst trains an encoder via MLM and then frozen the encoder to train the decoder only via two generative tasks.”
The Summary Loop: Learning to Write Abstractive Summaries Without Examples (2020, methodology): “Performance can further be improved by constraining the attention (Gehrmann et al., 2018; Gui et al., 2019; Wang et al., 2019) and using pretrained Transformer-based language models (Lewis et al., 2019; Chi et al., 2019; Edunov et al., 2019).”