Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

By Yiming Cui, Ziqing Yang, Xin Yao. 2023

426 citations recorded by the source corpus.

Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca} and Chinese Llama-2 series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca-2}

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ping Luo
2. Jianfeng Gao
3. Yu Qiao
4. Wenhai Wang
5. Tong Lu
6. Jifeng Dai
7. Qun Liu
8. Chunhua Shen

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yu Qiao
3. Ping Luo
4. Dacheng Tao
5. Xiaogang Wang
6. Chunhua Shen
7. Chen Change Loy
8. Alan L. Yuille

In Large Language Models

1. Qun Liu
2. Xin Jiang
3. Lifeng Shang
4. Yu Qiao
5. Peng Jin
6. Li Yuan
7. Xizhou Zhu
8. Jifeng Dai

Citations in the corpus

• CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility (2023, neutral): “7B (Tian et al., 2023), ChatPLUG-13B (Tian et al., 2023), MOSS (OpenLMLab, 2023), Chinese-AlpacaPlus-7B (Cui et al., 2023), Chinese-Alpaca-Plus13B (Cui et al., 2023), Ziya-LLaMA-13B (Zhang et al., 2022).”
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, methodology): “…Team, 2023; Sun et al., 2023; Du et al., 2021; Bai et al., 2023a; Yang et al., 2023; Penedo et al., 2023; Taori et al., 2023) with strong instruction-following and generalization capabilities have been applied to LVLMs. Early works such as BLIP-2 (Li et al., 2023b) and FROMAGe (Koh et al.,…”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “As shown in Figure 3, QLLaMA is developed based on the pre-trained multilingual LLaMA [26], and newly added 96 learnable queries and cross-attention layers (1 billion parameters) that are randomly initialized.”
• CMMLU: Measuring massive multitask language understanding in Chinese (2023, neutral): “…et al., 2022 2023), Bactrian-X-7B (Li et al., 2023a), Falcon-7B/40B (Almazrouei et al., 2023); (2) Chinese-oriented models: MOSS-SFT-16B (OpenLMLab, 2023), Chinese-LLaMA-7B/13B (Cui et al., 2023), Chinese-GLM-10B (Du et al., 2022), ChatGLM-6B (Zeng et al., 2023) and BatGPT-15B (Li et al., 2023b).”
• Baichuan 2: Open Large-scale Language Models (2023, neutral): “• Chinese-Alpaca-Plus-13B (Cui et al., 2023): A language model trained by fine-tuning LLaMA13B on the conversational dataset generated by ChatGPT.”
• Aligning Large Language Models with Human: A Survey (2023, methodology): “Several follow-up attempts, such as Aplaca (Taori et al., 2023) and its variants (Cui et al., 2023a) follow this Self-Instruct framework.”
• Sparsity-Accelerated Training for Large Language Models (2024, methodology): “By omitting computations related to some less important neurons, training on the right side proceeds faster compared to the left side, while maintaining similar performance. et al., 2023; Cui et al., 2023; Ke et al., 2023).”