GLM: General Language Model Pretraining with Autoregressive Blank Infilling

By Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang. 2021

1,956 citations recorded by the source corpus.

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25× parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Ping Luo
3. Yu Qiao
4. Mohit Bansal
5. Minlie Huang
6. Jie Tang
7. Luc Van Gool
8. Chunhua Shen

In Large Language Models

1. Minlie Huang
2. Jie Tang
3. Hui Zeng
4. Ming Ding
5. Xiao Liu
6. Yu Qiao
7. Ion Stoica
8. Ping Luo

Citations in the corpus

• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “A recent technical report (SenseTime, 2023) further evaluates foundation models (GLM130B (Du et al., 2022), LLaMa-65B (Touvron et al.”
• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “In GLM-130B, we instead make an attempt to explore the potential of a bidirectional GLM—General Language Model (Du et al., 2022)—as its backbone.”
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, background): “…Code Interpreter 2 are the most capable ones, (2) Gemini Ultra and Pro (Anil et al., 2023), (3) Inflection-2 (Inflection AI, 2023), (4) Grok-1 3 , as well as models recently released by Chinese companies including (5) Baichuan-3 4 , (6) the latest GLM-4 5 from the GLM family (Du et al., 2022).”
• Efficient Training of Language Models to Fill in the Middle (2022, positive): “Similar to our work, Zhu et al. [2019], Donahue et al. [2020], GLM [Du et al., 2022], CM3 [Aghajanyan et al., 2022], and InCoder [Fried et al., 2022] utilize left-to-right autoregressive modeling by moving the infill regions to the end of context, with regions separated by sentinels.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”
• Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, background): “Nowadays, large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chung et al., 2022; Chowdhery et al., 2022; Zhang et al., 2022; Touvron et al., 2023; Du et al., 2022) represented by ChatGPT2 have revolutionized the field of NLP by achieving remarkable results in a wide range of NLP ”
• DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “It is developed using the ChatGLM2 (Du et al., 2022) architecture and is enhanced with an extensive dataset of coding examples.”
• TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “Modern methods typically leverage off-the-shelf Large Language Models (LLMs) [5], [6] for scene descriptions to generate object affordance, location, or heatmaps, followed by a predefined motion planner to complete the tasks [7], [8].”
• Large Language Models in Finance: A Survey (2023, background): “Fine-tuning LLMs in the finance domain can enhance domain-specific language understanding and contextual comprehension, resulting in improved performance in finance-related tasks and generating more accurate and tailored outputs.”
• CMMLU: Measuring massive multitask language understanding in Chinese (2023, neutral): “…et al., 2022 2023), Bactrian-X-7B (Li et al., 2023a), Falcon-7B/40B (Almazrouei et al., 2023); (2) Chinese-oriented models: MOSS-SFT-16B (OpenLMLab, 2023), Chinese-LLaMA-7B/13B (Cui et al., 2023), Chinese-GLM-10B (Du et al., 2022), ChatGLM-6B (Zeng et al., 2023) and BatGPT-15B (Li et al., 2023b).”
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “Powerful LLMs (OpenAI, 2023; Touvron et al., 2023a; Wei et al., 2022; Touvron et al., 2023b; Zheng et al., 2023; Team, 2023; Sun et al., 2023; Du et al., 2021; Bai et al., 2023a; Yang et al., 2023; Penedo et al., 2023; Taori et al., 2023) with strong instruction-following and generalization…”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, background): “Encouraged by the rapid development of large language models (LLMs) [47, 4, 8, 9, 13, 1, 43], integrating visual modality into LLMs to enhance models’ interactivity capabilities has witnessed ever-changing advances in recent days [54, 26, 24, 11, 52, 2, 48, 31, 5, 12].”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “Our annotators carefully screened these videos using five well-defined criteria: • Clear Protagonist Presence : Videos must feature identifiable main characters (human or virtual) who appear consistently and interact meaningfully with their environment.”