All NLP Tasks Are Generation Tasks: A General Pretraining Framework

By Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang. 2021

68 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Jie Tang
2. Zhiyuan Liu
3. Minlie Huang
4. Xiao Liu
5. Ming Ding
6. Maosong Sun
7. Xu Han
8. Xipeng Qiu

In Natural Language Processing

1. Zhiyuan Liu
2. Maosong Sun
3. Xipeng Qiu
4. Xu Han
5. Yang Liu
6. Minlie Huang
7. Jie Tang
8. Ning Ding

Citations in the corpus

• CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “As an evidence, we used PB-relax successfully eliminating the overflow in training a 10B-parameter GLM [14].”
• Pre-Trained Models: Past, Present and Future (2021, methodology): “Taken from (Du et al., 2021).”
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “We include BERT-large (Devlin et al., 2018), RoBERTa-large (Liu et al., 2019), DeBERTa-xlarge (He et al., 2020), GLMxlarge/xxlarge (Du et al., 2021) for evaluation.”
• OpenPrompt: An Open-source Framework for Prompt-learning (2021, background): “Then the final prediction will be based on a mapping from the target texts to the labels (Ye et al., 2021; Du et al., 2021).”
• Towards a robust experimental framework and benchmark for lifelong language learning (2021, background): “Future work could investigate prompt-based learning as a means to expanding degree-of-belief to generation tasks through auto-regressive blank-filling of cloze-style question prompts [16].”
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022, background): “We include BERT-large (De-vlin et al., 2018), RoBERTa-large (Liu et al., 2019), DeBERTa-xlarge (He et al., 2020), GLM-xlarge/xxlarge (Du et al., 2021) for evaluation.”
• Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “UniLMv2 [5] and GLM [39] improved vanilla prefix masking strategy by introducing permuted language modeling in XLNet [198].”
• DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “Pretrained LMs (Devlin et al., 2019; Radford et al., 2019b; Yang et al., 2019) are the key ingredi-ents in contemporary NLP. Sequence-to-sequence (seq2seq) LMs target conditional generation, such as T5 (Raffel et al., 2019), BART (Lewis et al., 2020) and GLM (Du et al., 2021).”
• CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers (2022, methodology): “As shown in DebertaV2 [13], a large vocabulary (128,000 tokens) offers many benefits.”