GLM-130B: An Open Bilingual Pre-trained Model

By Aohan Zeng, Xiao Liu, Zhengxiao Du, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, W. Tam and 7 more. 2022

1,288 citations recorded by the source corpus.

We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B (davinci) on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B -- the largest Chinese language model -- across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization without post training, with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the most affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at \url{https://github.com/THUDM/GLM-130B/}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public.”
Large Language Models Are Human-Level Prompt Engineers (2022, background): “, 2021)) and one with reverse generation ability (INT4 quantized GLM-130B (Zeng et al., 2022)).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “(2020), with a few notable deviations based on recent advances in best practices for large scale language modeling (Black et al., 2022; Chowdhery et al., 2022; Zeng et al., 2022):”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “Lossy quantization approaches study the trade-offs for regular rounding [13, 72, 48] or how to optimize rounding decisions to improve quantization precision [18].”
StarCoder: may the source be with you! (2023, methodology): “Tsinghua University released the weights of GLM-130B (Zeng et al., 2022), a Chinese-English LLM, and CodeGeeX-13B (Zheng et al., 2023), a LLM for coding applications, without releasing the training sets.”
Stable and low-precision training for large-scale vision-language models (2023, background): “Another line of work studies large models inference with more than 250M parameters by considering 16-bit inputs and k-bit weights [16, 24, 68].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “While it has been shown that it is sufficient to use 16-bit inputs and 8-bit weights to avoid this disruption (Zeng et al., 2022), it is unclear if outlier features can cause degradation if we use 16-bit inputs and weights below 8-bit precision.”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “Experiments show that SmoothQuant is hardware-efficient: it can maintain the performance of OPT-175B (Zhang et al., 2022), BLOOM-176B (Scao et al., 2022) , GLM-130B (Zeng et al., 2022), and MT-NLG 530B (Smith et al., 2022), leading to up to 1.51 × speed up and 1.96 × memory saving on PyTorch.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Another parallel work is GLM-130B which uses insights from our work to achieve zero-degradation 8-bit quantization (Zeng et al., 2022).”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “GLM-130B (Zeng et al., 2023) and ChatGLM-6B (THUDM, 2023) are based on the General Language Model architecture (GLM, Du et al.”
What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “Last two rows show that our multilingual model matches the XGLM results. the least frequent ones, so that all languages are represented.”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Large language models (LLMs) have garnered significant attention due to their exceptional capabilities in text understanding and generation [23, 31].”
MemoryBank: Enhancing Large Language Models with Long-Term Memory (2023, methodology): “(Zeng et al., 2022) and Alpaca (Taori et al., 2023).”
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face (2023, background): “In recent years, the ﬁeld of natural language processing (NLP) has been revolutionized by the emergence of large language models (LLMs) [1, 2, 3, 4, 5, 19, 6], exempliﬁed by models such as GPT-3 [1], GPT-4 [20], PaLM [3], and LLaMa [6].”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, background): “The recent open-sourcing of LLMs [70–73, 100] further accelerates the emergence of public MLLMs [ 21, 47, 104].”