VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation

By Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, Luo Si. 2020

23 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “, 2020) and VECO (Luo et al., 2020) benefit from parallel training data, while X-STILTs (Phang et al.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “VECO (Luo et al., 2020) utilizes cross-attention MLM to pretrain a variable cross-lingual language model for both NLU and NLG. mT6 (Chi et al., 2021a) improves mT5 (Xue et al., 2021) by learning the translation span corruption task on parallel data.”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, background): “There are also other work (Huang et al., 2019; Luo et al., 2020) to explore different pretraining tasks to further improve the performance of pretrained multilingual model.”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, methodology): “Following Luo et al. 2020, we used the averaged representation in the middle layer of the best XNLI model to evaluate the retrieval task.”
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, background): “Recent works have shown that cross-lingual transfer learning in pre-trained multilingual models such as mBERT (Devlin et al., 2019) and XLM-R (Con-neau et al., 2020) could be improved further by us-ing parallel data (Conneau and Lample, 2019; Hu et al., 2020a; Ouyang et al., 2020; Luo et al., 2020).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “, 2020a) to some of the more recent models like VECO (Luo et al., 2021).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “VECO (Luo et al., 2020) is a 24-layer encoder-decoder model that uses additional MLM variants during pre-training.”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “VECO (Luo et al., 2020) pretrains a variable cross-lingual pre-training model that learns uni-”