Baichuan 2: Open Large-scale Language Models

By Ai Ming Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan and 43 more. arXiv.org, 2023

1,014 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c96297261467b5daa2d01227496a70d444602434

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Hongsheng Li
3. Ping Luo
4. Peng Gao
5. Jifeng Dai
6. Tong Lu
7. Wenhai Wang
8. Renrui Zhang

Citations in the corpus

• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4, 5] promises vast transformative potential across multiple sectors, from healthcare [6, 7, 8] and education [9, 10, 11] to robotics [12, 13, 14] and commerce [15].”
• Large Language Models for Information Retrieval: A Survey (2023, neutral): “0 [68, 114], Baichuan [115], and Qwen [116], specifically for the query rewriting task.”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “The progress in the field has been further accelerated by the emergence of open-source LLMs, including the LLaMA series [106, 107], Vicuna [145], InternLM [104], MOSS [101], ChatGLM [36], Qwen [4], Baichuan [6], and Falcon [86], among others [26, 103, 119].”
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, neutral): “This method updates each token once using the first-order Taylor approximation around the current token embedding’s gradient with respect to the target loss.”
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025, neutral): “The emergence of AI Agent as a transformative paradigm in artificial intelligence is closely tied to the evolution and repurposing of large-scale language models such as GPT-3 [120], Llama [121], T5 [122], Baichuan 2 [123] and GPT3mix [124].”
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “…manner [Ouyang et al. 2022; OpenAI 2022, 2023b; Google 2023; Anthropic 2023; Workshop et al. 2022; Touvron et al. 2023a,b; Bai et al. 2023a; Yang et al. 2023; Team 2023; Bi et al. 2024] and incredible scalability [Kaplan et al. 2020; Henighan et al. 2020; Hoffmann et al. 2022; Wei et al.…”
• GraphGPT: Graph Instruction Tuning for Large Language Models (2023, neutral): “This significant reduction in token consumption translates to a substantial decrease in training and inference resource requirements.”
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “Other notable LLMs encompass a range of developments, such as the OPT [96], LLaMA series [78, 79], e.g ., Mistral [37, 38], InternLM [77], GLM series [25, 93], Qwen series [6, 65], Baichuan [7], Skywork [84] and Falcon [61] have made significant contributions to the field.”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, background): “As large language models (LLMs) [8, 43, 43, 47, 42, 34, 36, 9] rapidly advance, a growing fraction of the research community is focusing on integrating visual content into LLMs to build a powerful intelligent assistant with more interactive ways.”
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, background): “Recent breakthroughs in artificial intelligence have been driven notably by the development of large language models (LLMs) [2, 4, 8, 9, 12, 53, 56].”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “However, this method disregards the overall LLM structure, resulting in an irregular sparse model composition.”
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, positive): “Concurrently, models such as Qwen (Bai et al., 2023), Baichuan (Yang et al., 2023a), and InternLM (Team, 2023) have advanced bilingual LLM capabilities, whereas TinyLlama (Zhang et al., 2024b) and Phi-2 (Microsoft, 2023) focus on reducing parameters for edge deployment.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”